5 p.m.
Hello,
I am ready to begin defining requirements for an audit event extraction and
parsing library. The purpose is two-fold, it should have the ability to
extract records from a given file and it must have the ability to parse a
buffer handed to it. This will allow audit event dispatcher utilities to
process events, too. The library should also be done in a way that is
friendly to other languages like python.
Comments?
-Steve
8:14 a.m.
Steve Grubb <sgrubb(a)redhat.com> writes:
I am ready to begin defining requirements for an audit event
extraction and parsing library.
One aspect of such a library is to define the structure of events as
seen by consumers of the information. From reading the manual page,
and observing output, I conclude ausearch produces a sequence of
events. Each event is a sequence of tables, one table is specified on
one line of output. A table is a sequence of key-value pairs, and
sometimes the value is missing.
If this description accurately reflects the structure of the output of
ausearch, a simple callback-based programming interface is suggested.
For example, a consumer of the information could define a structure of
type struct audit_event_callback, and then pass a pointer to it to the
provider of audit data, in the following example, the gimme function.
typedef struct audit_event_callback {
void (*end_event)(void); /* Note the end of an event. */
void (*end_table)(void); /* Note the end of a table in an event. */
/* Note one entry in a table in an event--a key-value pair. */
/* The value may be NULL. */
void (*entry)(const char *key, const char *value);
} *audit_event_callback_t;
int gimme(audit_event_callback_t a_callback/* other input */) {
/* ...*/ return 0;
}
With this kind of interface, if an audit event consumer passed the
pointer aec as its callback to the gimme function for data ausearch
would output as:
----
type=SYSCALL syscall=umount2
type=PATH name=/media/LEXAR_MEDIA
the following calls would be produced.
aec->entry("type", "SYSCALL");
aec->entry("syscall", "umount2");
aec->end_table();
aec->entry("type", "PATH");
aec->entry("name", "/media/LEXAR_MEDIA");
aec->end_table();
aec->end_event();
On another topic, I would like to see the library interface commit to
providing only US-ASCII strings. This way, issues of character sets
can be avoided.
John
12:53 p.m.
Steve Grubb <sgrubb(a)redhat.com> writes:
I am ready to begin defining requirements for an audit event
extraction and parsing library.
One aspect of such a library is to define the structure of events as
seen by consumers of the information. From reading the manual page,
and observing output, I conclude ausearch produces a sequence of
events. Each event is a sequence of tables, one table is specified on
one line of output. A table is a sequence of key-value pairs, and
sometimes the value is missing.
If this description accurately reflects the structure of the output of
ausearch, a simple callback-based programming interface is suggested.
For example, a consumer of the information could define a structure of
type struct audit_event_callback, and then pass a pointer to it to the
provider of audit data, in the following example, the gimme function.
typedef struct audit_event_callback {
void (*end_event)(void); /* Note the end of an event. */
void (*end_table)(void); /* Note the end of a table in an event. */
/* Note one entry in a table in an event--a key-value pair. */
/* The value may be NULL. */
void (*entry)(const char *key, const char *value);
} *audit_event_callback_t;
int gimme(audit_event_callback_t a_callback/* other input */) {
/* ...*/ return 0;
}
With this kind of interface, if an audit event consumer passed the
pointer aec as its callback to the gimme function for data ausearch
would output as:
----
type=SYSCALL syscall=umount2
type=PATH name=/media/LEXAR_MEDIA
the following calls would be produced.
aec->entry("type", "SYSCALL");
aec->entry("syscall", "umount2");
aec->end_table();
aec->entry("type", "PATH");
aec->entry("name", "/media/LEXAR_MEDIA");
aec->end_table();
aec->end_event();
On another topic, I would like to see the library interface commit to
providing only US-ASCII strings. This way, issues of character sets
can be avoided.
John
1:04 p.m.
On Tuesday 07 March 2006 13:53, John D. Ramsdell wrote:
From reading the manual page, and observing output, I conclude
ausearch
produces a sequence of events. Each event is a sequence of tables, one
table is specified on one line of output. A table is a sequence of
key-value pairs, and sometimes the value is missing.
The value should never be missing. Yes, each event is composed of individual
records which have fields. They are normalized (except for SE Linux AVC
messages).
If this description accurately reflects the structure of the output
of
ausearch, a simple callback-based programming interface is suggested.
For example, a consumer of the information could define a structure of
type struct audit_event_callback, and then pass a pointer to it to the
provider of audit data, in the following example, the gimme function.
We could have both a linked list representation and smart iterator. In the
latter case, you could have a next_event, next_record, and next_field
functions that walk that list. I also think we want a reset(), find_field(),
find_field_next() function to reset internal pointers,find a particular key,
and find the next instance of that key within the same event.
Are there any issues that need to be taken into account for Python
compatibility?
Debbie, did you have any comments?
-Steve
2:13 p.m.
Steve Grubb <sgrubb(a)redhat.com> writes:
Are there any issues that need to be taken into account for Python
compatibility?
When information flows only from the audit search program to a Python
program, one can connect the two via a pipe, and transmit an XML
document. The following DTD captures the structure Steve described,
and the C code needed to generate data in this format is trivial.
<!ELEMENT au (seq)*>
<!ELEMENT seq (tab)+>
<!ELEMENT tab (ent)+>
<!ELEMENT ent EMPTY>
<!ATTLIST ent
key CDATA #REQUIRED
val CDATA #REQUIRED>
For Python programs that dynamically link in the audit parsing
library, one would like to use a C interface that maps well to
the Python/C API described at:
http://docs.python.org/api/api.html
I haven't used this API in a while, as I've been embedding Lua into my
C applications when I want to extend a C program with internal
scripting. I'll read the API carefully, and get back to you with any
issues I discover.
If we want to be scripting language neutral, we should use SWIG.
http://www.swig.org
Once again, I haven't used this in a long time, but I'll take a look
at it. I know functions in the libsemanage interface are made
available to multiple languages via SWIG. We could ask Tresys for
advice.
John
I end with output from the tool I currently use to reformat ausearch
output.
<?xml version="1.0"?>
<!DOCTYPE au [
<!ELEMENT au (seq)*>
<!ELEMENT seq (tab)+>
<!ELEMENT tab (ent)+>
<!ELEMENT ent EMPTY>
<!ATTLIST ent
key CDATA #REQUIRED
val CDATA #IMPLIED>
]>
<au>
<seq>
<tab>
<ent key="type" val="PATH"/>
<ent key="msg" val="audit(03/07/2006 12:18:03.698:18)"/>
<ent key=":"/>
<ent key="item" val="1"/>
<ent key="name" val="(null)"/>
<ent key="inode" val="17284616"/>
<ent key="dev" val="08:01"/>
<ent key="mode" val="file,755"/>
<ent key="ouid" val="root"/>
<ent key="ogid" val="root"/>
<ent key="rdev" val="00:00"/>
<ent key="obj" val="system_u:object_r:ld_so_t:s0"/>
</tab>
<tab>
<ent key="type" val="PATH"/>
<ent key="msg" val="audit(03/07/2006 12:18:03.698:18)"/>
<ent key=":"/>
<ent key="item" val="0"/>
<ent key="name" val="/bin/ls"/>
<ent key="inode" val="6678183"/>
<ent key="dev" val="08:01"/>
<ent key="mode" val="file,755"/>
<ent key="ouid" val="root"/>
<ent key="ogid" val="root"/>
<ent key="rdev" val="00:00"/>
<ent key="obj" val="system_u:object_r:ls_exec_t:s0"/>
</tab>
<tab>
<ent key="type" val="CWD"/>
<ent key="msg" val="audit(03/07/2006 12:18:03.698:18)"/>
<ent key=":"/>
<ent key="cwd" val="/home/bsniffen"/>
</tab>
<tab>
<ent key="type" val="SYSCALL"/>
<ent key="msg" val="audit(03/07/2006 12:18:03.698:18)"/>
<ent key=":"/>
<ent key="arch" val="i386"/>
<ent key="syscall" val="execve"/>
<ent key="success" val="yes"/>
<ent key="exit" val="0"/>
<ent key="a0" val="bfa05bd1"/>
<ent key="a1" val="bfa04408"/>
<ent key="a2" val="bfa04414"/>
<ent key="a3" val="bfa04408"/>
<ent key="items" val="2"/>
<ent key="pid" val="2202"/>
<ent key="auid" val="bsniffen"/>
<ent key="uid" val="root"/>
<ent key="gid" val="root"/>
<ent key="euid" val="root"/>
<ent key="suid" val="root"/>
<ent key="fsuid" val="root"/>
<ent key="egid" val="root"/>
<ent key="sgid" val="root"/>
<ent key="fsgid" val="root"/>
<ent key="tty" val="pts0"/>
<ent key="comm" val="ls"/>
<ent key="exe" val="/bin/ls"/>
<ent key="subj"
val="user_u:system_r:unconfined_t:s0-s0:c0.c255"/>
</tab>
</seq>
</au>
2:20 p.m.
On Tuesday 07 March 2006 15:13, John D. Ramsdell wrote:
When information flows only from the audit search program to a
Python
program, one can connect the two via a pipe,
At this point, I am thinking about library API.
If we want to be scripting language neutral, we should use SWIG.
This is what we do for audit-libs-python package. I was planning to extend
that to accommodate the new library. My real question is can Python use
structures returned by C? Or does it need data broken down into ints and
chars?
-Steve
4:23 p.m.
Below are the structures that we (Loulwa Salem, Mike Thompson, Tim Chavez
and I) had envisioned the structures for the new API to look like.
Basically we imagine a list of lists. Below that are some function
prototypes needed in the API.
I'm new to Python so I will study the Python/C API link posted.
typedef struct records {
time_t time;
serial_t serial;
record_t record;
records_t * next_record;
} records_t;
typedef struct record {
recordtype kind; //what kind of audit_record record it is
record_data data;
record * next;
} record_t;
union record_data {
syscall_t syscall;
ipc_t ipc;
fs_watch_t fs_watch;
fs_inode_t fs_inode;
cwd_t cwd;
path_t path;
config_change_t config_change;
daemon_end_t daemon_end;
user_t user;
user_chauthtok_t user_chauthtok;
user_auth_t user_auth;
user_acct_t user_acct;
user_start_t user_start;
user_end_t user_end;
cred_acq_t cred_acq;
cred_disp_t cred_disp;
cred_refr_t cred_refr;
login_t longin;
} record_data_t;
typedef struct syscall {
arch_t arch;
syscall_num_t syscall_num; //note used syscall_num rather than
syscall at some point they could add syscall_name
success_t success;
exit_t exit;
a_t a0;
a_t a1;
a_t a2;
a_t a3;
items_t items;
id_t pid;
id_t auid;
id_t uid;
id_t gid;
id_t euid;
id_t suid;
id_t fsuid;
id_t egid;
id_t sgid;
id_t fsgid;
comm_t comm;
exe_t exe;
} syscall_t;
typedef struct fs_watch {
inode_t watch_inode;
watch_t watch;
filterkey_t filterkey;
perm_t perm;
perm_mask_t perm_mask;
} fs_watch_t;
typedef struct fs_inode {
inode_t inode;
id_t inode_uid;
id_t inode_gid;
dev_t inode_dev;
rdev_t inode_rdev
} fs_inode_t;
typedef struct cwd {
cwd_t cwd;
} cwd_t;
typedef struct path {
name_t name;
flags_t flags;
inode_t inode;
dev_t dev;
mode_t mode;
id_t ouid;
id_t ogid;
rdev_t rdev;
} path_t;
typedef struct config_change {
time_t time;
id_t auid;
text_t text; //"removed watch"
audit_enabled_t audit_enabled;
old_t old;
} config_change_t;
typedef struct daemon_start {
version_t version;
format_t format;
id_t auid;
id_t auditd_pid;
text_t text; //"auditd start"
} daemon_start_t;
typedef struct daemon_end {
id_t auid;
id_t sending_pid;
id_t auditd_pid;
text_t text; //"auditd normal halt"
} daemon_end_t;
typedef struct ipc {
qbytes_t qbytes;
id_t iuid;
id_t igid;
mode_t mode;
} ipc_t;
typedef struct login {
msg_t msg; //message needs to be parsed to figure out the
following
id_t pid;
id_t uid;
id_t old_auid;
id_t new_auid;
} login_t;
typedef struct user {
id_t pid;
id_t uid;
id_t auid;
msg_t msg;
text_t text; //"user"
} user_t;
typedef struct user_chauthtok {
id_t pid;
id_t uid;
id_t auid;
msg_t msg; //message needs to be parsed to figure out the
following
user_t user;
exe_t exe;
hostname_t hostname;
addr_t addr;
terminal_t terminal;
res_t res; //records using 'res' should be changed to
use 'result'
result_t result;
op_t op;
acct_t acct;
} user_chauthtok_t;
Note: user_auth, user_acct, user_start, user_end, cred_acq, cred_disp,
cred_refr have the same tydef, but the message parsed differs slightly.
typedef struct user_auth {
id_t pid;
id_t uid;
id_t auid;
msg_t msg; //message needs to be parsed to figure out the
following
user_t user;
exe_t exe;
hostname_t hostname;
addr_t addr;
terminal_t terminal;
result_t result;
} user_auth_t;
typedef struct user_acct {
id_t pid;
id_t uid;
id_t auid;
msg_t msg; //message needs to be parsed to figure out the
following
user_t user;
exe_t exe;
hostname_t hostname;
addr_t addr;
terminal_t terminal;
result_t result;
} user_acct_t;
typedef struct user_start {
id_t pid;
id_t uid;
id_t auid;
msg_t msg; //message needs to be parsed to figure out the
following
user_t user;
exe_t exe;
hostname_t hostname;
addr_t addr;
terminal_t terminal;
result_t result;
} user_start_t;
typedef struct user_end {
id_t pid;
id_t uid;
id_t auid;
msg_t msg; //message needs to be parsed to figure out the
following
user_t user;
exe_t exe;
hostname_t hostname;
addr_t addr;
terminal_t terminal;
result_t result;
} user_end_t;
typedef struct cred_acq {
id_t pid;
id_t uid;
id_t auid;
msg_t msg; //message needs to be parsed to figure out the
following
user_t user;
exe_t exe;
hostname_t hostname;
addr_t addr;
terminal_t terminal;
result_t result;
} cred_acq_t;
typedef struct cred_disp {
id_t pid;
id_t uid;
id_t auid;
msg_t msg; //message needs to be parsed to figure out the
following
user_t user;
exe_t exe;
hostname_t hostname;
addr_t addr;
terminal_t terminal;
result_t result;
} cred_disp_t;
typedef struct cred_refr {
id_t pid;
id_t uid;
id_t auid;
msg_t msg; //message needs to be parsed to figure out the
following
user_t user;
exe_t exe;
hostname_t hostname;
addr_t addr;
terminal_t terminal;
result_t result;
} cred_refr_t;
/* The following struct is the generic record struct which does not have a
record type */
/* It is used by get_by_fields(). */
typedef struct generic_field_data {
a_t a0;
a_t a1;
a_t a2;
a_t a3;
acct_t acct;
addr_t addr;
arch_t arch;
comm_t comm;
cwd_t cwd;
dev_t dev;
dev_t inode_dev;
audit_enabled_t audit_enabled;
exe_t exe;
exit_t exit;
filterkey_t filterkey;
flags_t flags;
format_t format;
hostname_t hostname;
id_t auditd_pid;
id_t auid;
id_t egid;
id_t euid;
id_t fsgid;
id_t fsuid;
id_t gid;
id_t igid;
id_t inode_gid;
id_t inode_uid;
id_t iuid;
id_t new_auid;
id_t ogid;
id_t ouid;
id_t old_auid;
id_t ouid;
id_t pid;
id_t sending_pid;
id_t sgid;
id_t suid;
id_t uid;
inode_t inode;
inode_t watch_inode;
items_t items;
mode_t mode;
msg_t msg;
name_t name;
old_t old;
op_t op;
perm_mask_t perm_mask;
perm_t perm;
qbytes_t qbytes;
rdev_t inode_rdev;
rdev_t rdev;
result_t res;
result_t result;
serial_t serial;
success_t success;
syscall_num_t syscall_num; //note used syscall_num rather than
syscall at some point they could add syscall_name
terminal_t terminal;
text_t text; // example "removed watch"
time_t time;
type_t type;
user_t user;
version_t version;
watch_t watch;
} generic_field_data_t;
Some function prototypes:
struct record * get_records(time_t start, time_t end){
// start and end time can be specified
// If zero is passed in for start time, then start searching at beginning
of log
// If zero is passed in for the end time, then search until the end of the
log
// returns pointer to link list of all records within start and end time
(which are themselves a link list)
// if no records were found during specified start and end time, returns
NULL
}
struct record * get_matching_records(struct record * matching_criteria,
time_t start, time_t end){
// matching_criteria is passed into function
// matching_criteria is a pointer to a linked list of record pieces we are
looking for
// start and end time can be specified
// If zero is passed in for start time, then start searching at beginning
of log
// If zero is passed in for the end time, then search until the end of the
log
// returns pointer to link list of all matching records (which are
themselves a link list)
// if no matching record is found, returns NULL
}
struct record * get_records_by_fields(struct generic_field_data *
fields_to_match, time_t start, time_t end){
// fields_to_match is passed into the function
// fields_to_match is a pointer to one structure with all the fields we
want to match filled in and the rest initialized
// start and end time can be specified
// If zero is passed in for start time, then start searching at beginning
of log
// If zero is passed in for the end time, then search until the end of the
log
// returns pointer to link list of all matching records (which are
themselves a link list)
// if no matching record is found, returns NULL
}
record_data * initialize_record_data(record_data * matching_criteria){
//initializes all fields in generic field data type
}
generic_field_data * initialize_generic_field_data(generic_field_data *
fields_to_match){
//initializes all fields in generic field data type
}
8:10 p.m.
On Tue, Mar 07, 2006 at 03:23:41PM -0700, Debora Velarde wrote:
Below are the structures that we (Loulwa Salem, Mike Thompson, Tim
Chavez
and I) had envisioned the structures for the new API to look like.
Basically we imagine a list of lists. Below that are some function
prototypes needed in the API.
[...]
typedef struct syscall {
arch_t arch;
syscall_num_t syscall_num;
success_t success;
exit_t exit;
a_t a0;
a_t a1;
I think using C structures for each type of entry is a maintenance
nightmare, and you'd need to recompile code using them each time a
definition changes even if you don't care about any of the added fields.
Wouldn't it be much easier to conceptually treat the audit records as
hashes (collections of tag/value pairs), where the tags are strings and
the values are either also strings (for a very low-level interface), or
explicitly typed objects such as integers, strings, lists, etc.?
Something like this:
log = audit_log_open(file);
while ( (record = audit_get_record(log)) != NULL) {
if audit_record_match(record, "type", "syscall") {
/* get a typed object */
item_t item = audit_record_get(record, "a0");
if (item->type == T_INT) foo+=item->intval;
/* assume that a type is numeric, lib will
* throw an errow if that's not correct */
int num = audit_record_get_int(record, "syscall_num");
/* if you don't care about the type, get a string */
char *success = audit_record_get_string(record, "success");
} else {
/* loop over all entries */
iter = audit_item_iter(record);
item_t item;
while ( (item = audit_next_item(iter) != NULL) {
/* ... */
}
}
}
An iterator-based approach should be easy to use from Python as well.
If you worry about the efficiency or type-safety of string-based hash
references, abstract that out so that it's optimizable later. Something
like:
tag_t syscall_num = audit_get_tag("syscall");
for (...)
n = audit_record_get(record, syscall_num);
(This approach is somewhat inspired by Lisp-style SEXP lists using
interned symbols and type-tagged data. Hey, it worked for 40 years, why
invent something new?)
-Klaus
8:36 p.m.
On Tue, Mar 07, 2006 at 08:10:02PM -0600, Klaus Weidner wrote:
An iterator-based approach should be easy to use from Python as well.
This would be especially nice if you export the typed objects from the
audit log as python objects of the correct type, then something like
this should work:
import auditlog
log = auditlog.open(file)
for record in log:
if record.type == auditlog.SYSCALL:
print record.isodate,
print "call %d, arg0=%d\n" %
(record.syscall_num, record.a0)
else:
d = record.dict()
for tag in d:
print tag, d[tag]
Similarily for other high-level languages that support dynamic typing.
In general, I think that the interface should have the following
properties:
- code that expects certain fields to be present will continue to work
when new fields get added (and will ignore them)
- if expected fields are absent, you get a (trappable) error
- if code loops over all fields and handles each item in a type-safe way,
it'll support new fields automatically with no code change when they
are added
-Klaus
6:11 a.m.
Klaus Weidner <klaus(a)atsec.com> writes:
Wouldn't it be much easier to conceptually treat the audit
records
as hashes (collections of tag/value pairs), ...
I agree.
I was thinking about why I proposed a callback-based interface, and
Steve countered with a iterator based-interface. I don't think this
difference is simply a matter of style, but rather a matter of
expected usage.
I would like a library interface that allows me to reuse existing
functionality in ausearch, not just process raw records. Thus, I
imagined at least two functions being part of the library.
int from_stream(audit_event_callback_t aec, FILE *source);
int ausearch(audit_event_callback_t aec /* ausearch options */);
where
typedef struct audit_event_callback {
void (*end_event)(void); /* Note the end of an event. */
void (*end_table)(void); /* Note the end of a table in an event. */
/* Note one entry in a table in an event--a key-value pair. */
void (*entry)(const char *key, const char *value);
} *audit_event_callback_t;
The arguments to the ausearch function would mirror the information
derived from the command-line arguments of the ausearch program. In
fact, in this design, the ausearch program would simply convert its
command-line arguments into C data types, create a callback that
produces at textual representation of the search results, and then
return the results of calling the ausearch library function with
these parameters.
I should explain my immediate interest is audit output. I'm working
on an SELinux policy generation tool called Polgen that uses system
call traces as one of its sources of information. Polgen currently
gathers this information using a modified version of strace. I'm
looking for output that mimics strace, so I assume I want system call
records that have been interpreted, as is done by ausearch using the
-i option. I'd like to easily use the record interpretation facility
already built into ausearch, along with its other fine features.
John
9:39 a.m.
On Wednesday 08 March 2006 07:11, John D. Ramsdell wrote:
I would like a library interface that allows me to reuse existing
functionality in ausearch, not just process raw records. Thus, I
imagined at least two functions being part of the library.
That is why I countered with an iterator interface. Ausearch iterates through
the records and would be easy to adapt it and aureport to such a library
interface.
I have also been thinking about key/value pair as the representation for
exactly the same reason Klaus mentioned. It should be extensible and a
minimal amount of maintenance. Key/value interface should allow Python to
access anything in case it does not understand "C" structures.
I'll take a hack at proposing an API and send it in a little while.
-Steve
10:28 a.m.
Steve Grubb <sgrubb(a)redhat.com> writes:
I'll take a hack at proposing an API and send it in a little
while.
Cool.
I looked at both the SWIG and Python/C API, and then realized that I
sit just two doors down from Sam Bayer. He has made extensive use of
these tools, and here is what I learned from talking with him.
(1) Even if you plan to only support Python as the scripting language,
use SWIG to generate as much of the interface as you can, because
it automatically generates a lot of Python/C API code for you.
(2) Even if you plan to specify your interfaces only in SWIG, become
familiar with the Python/C API so you can read the generated
code. Sam found important errors by carefully reading generated
code. Memory management issues become obvious by reading the code.
(3) It is very simple to use SWIG if the only objects that are passed
across the language boundary are numbers, strings, and opaque
pointers.
(4) Sam avoided passing structures, preferring to define classes in
pure Python code, and using a function interface to implement
methods of the class. The SWIG document suggests structures can
be used, and appear as a class in Python. I think Sam likes the
idea of a thin interface, and passing a structure is too
heavy-weight for his tastes.
(5) One can pass functions in both directions, and Sam described all
the details to me that is involved with passing and invoking a
Python function in C. In this case, one cannot use SWIG. My
reading is Steve is unlikely to define an interface that passes
either C or Python functions, so these difficulties can be
ignored.
So read the SWIG 1.3 Basics Chapter, and then flip to the chapter on
Python, and along with the Python/C API document, you should be able
access the information you need.
John
10:06 a.m.
On Wednesday 08 March 2006 10:39, Steve Grubb wrote:
I'll take a hack at proposing an API and send it in a little
while.
OK, here's what I have:
The audit library parser could have the following functions:
auparse_init - allow init of library. Set data source: logs, file, buffer.
ausearch_set_param - set search options
ausearch_next_event - traverse to the next event that yields a match based on
search criteria.
auparse_next_event - traverse to next event. This allows access to time and
serial number.
auparse_get_time - retrieve time stamp of current record
auparse_get_serial - retrieve serial number of current record
auparse_first_record - set iterator to first record in current event
auparse_next_record - traverse to next record in event. This allows access to
the event type
auparse_get_type - retrieve type of current record
auparse_first_field - set field pointer to first in current record
auparse_next_field - traverse the fields in a record
auparse_find_field() - find a given field in a event or record
auparse_find_field_next() - find the next occurance of that field in the same
record
auparse_get_field_str - return current field value as a string
auparse_get_field_int - return current field value as an int
auparse_interpret_field - interpret the current field as a string
auparse_destroy - free all data structures and close file descriptors
This would allow the following kind of programming:
auparse_init
ausearch_set_param
while ausearch_next_event
if auparse_find_field
auparse_interpret_field
print out
...
auparse_destroy
This is essentially how ausearch works.
The data structures would be hidden from the external application. Access to
fields is a name/value style. You access the fields through functions that
either return str pointer or ints.
Would something like this meet everyone's needs?
-Steve
10:48 a.m.
linux-audit-bounces(a)redhat.com wrote on 03/09/2006 10:06:47 AM:
On Wednesday 08 March 2006 10:39, Steve Grubb wrote:
> I'll take a hack at proposing an API and send it in a little while.
OK, here's what I have:
The audit library parser could have the following functions:
auparse_init - allow init of library. Set data source: logs, file,
buffer.
ausearch_set_param - set search options
What sort of search options can you set? Are these basically the same that
ausearch allows? If so, being able to search based on the value of any
field would be useful for testing (although they might not be in a
real-world application, e.g. a0).
ausearch_next_event - traverse to the next event that yields a match
based on
search criteria.
auparse_next_event - traverse to next event. This allows access to time
and
serial number.
auparse_get_time - retrieve time stamp of current record
auparse_get_serial - retrieve serial number of current record
auparse_first_record - set iterator to first record in current event
auparse_next_record - traverse to next record in event. This allows
access to
the event type
auparse_get_type - retrieve type of current record
auparse_first_field - set field pointer to first in current record
auparse_next_field - traverse the fields in a record
auparse_find_field() - find a given field in a event or record
auparse_find_field_next() - find the next occurance of that field inthe
same
record
auparse_get_field_str - return current field value as a string
auparse_get_field_int - return current field value as an int
auparse_interpret_field - interpret the current field as a string
auparse_destroy - free all data structures and close file descriptors
This would allow the following kind of programming:
auparse_init
ausearch_set_param
while ausearch_next_event
if auparse_find_field
auparse_interpret_field
print out
...
auparse_destroy
This is essentially how ausearch works.
The data structures would be hidden from the external application.
Access to
fields is a name/value style. You access the fields through functions
that
either return str pointer or ints.
Would something like this meet everyone's needs?
-Steve
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
12:59 p.m.
On Thursday 09 March 2006 11:48, Michael C Thompson wrote:
What sort of search options can you set?
Oh...how about any?
Are these basically the same that ausearch allows?
We could limit it to that and it would be faster...or it could be more. Do we
need to search on all field or just have access to all fields?
-Steve
1:13 p.m.
Steve Grubb <sgrubb(a)redhat.com> wrote on 03/09/2006 12:59:01 PM:
On Thursday 09 March 2006 11:48, Michael C Thompson wrote:
> What sort of search options can you set?
Oh...how about any?
Any - good to know. Sometimes people have mental models which incorperate
limits which aren't clearly communicated, hence the question.
> Are these basically the same that ausearch allows?
We could limit it to that and it would be faster...or it could be more.
Do we
need to search on all field or just have access to all fields?
From a testing/tester standpoint, if I understand ausearc's
purpose
correctly, it would be nice to be able to specify and field/value pair. My
understanding of auserach is to set up our search paramters (through many
tedious function calls! <humor, ignore it if you don't find funny>), and
then call ausearch_next_event to begin returning records which match the
parameters we've set. If this is the case, from a testing standpoint, it
would be nice to be able to set up the parameters on every value of the
record as we expect it to look. Obviously, if this is deemed unreasonable,
there are other ways to do this type of checking, but if it could be
incoperated right into the library, it would decrease complexity for the
caller if this is their objective.
Mike
-Steve
1:21 p.m.
On Thursday 09 March 2006 14:13, Michael C Thompson wrote:
My understanding of auserach is to set up our search paramters and
then call ausearch_next_event to begin returning records which match the
parameters we've set.
Yes.
If this is the case, from a testing standpoint, it would be nice to
be able
to set up the parameters on every value of the record as we expect it to
look.
Please give an example to make sure I understand you.
-Steve
3:16 p.m.
Steve Grubb <sgrubb(a)redhat.com> wrote on 03/09/2006 01:21:12 PM:
On Thursday 09 March 2006 14:13, Michael C Thompson wrote:
> My understanding of auserach is to set up our search paramters and
> then call ausearch_next_event to begin returning records which match
the
> parameters we've set.
Yes.
> If this is the case, from a testing standpoint, it would be nice to be
able
> to set up the parameters on every value of the record as we
expect it
to
> look.
Please give an example to make sure I understand you.
Example:
chmod("myfile", 0777)
I would like to be able to specify to ausearch_set_param("a1", 0777). I
did not see in my version of ausearch that it was possible to specify an
argument field for a field/value pair. Again, on top of this,
ausearch_set_param("a2",0) would be a good thing to add to ensure, as you
expressed in last Monday's open LSPP call, that you would like regression
tests run to ensure there was no uncleared context lying around.
Does this help?
Mike
-Steve
11:03 a.m.
If I want to match on two params (say syscall name and group id) would I
call ausearch_set_param twice or pass ausearch_set_param all my parameters
in one call? Can you post how you imagine the call to look like?
linux-audit-bounces(a)redhat.com wrote on 03/09/2006 08:06:47 AM:
On Wednesday 08 March 2006 10:39, Steve Grubb wrote:
> I'll take a hack at proposing an API and send it in a little while.
OK, here's what I have:
The audit library parser could have the following functions:
auparse_init - allow init of library. Set data source: logs, file,
buffer.
ausearch_set_param - set search options
ausearch_next_event - traverse to the next event that yields a match
based on
search criteria.
auparse_next_event - traverse to next event. This allows access to time
and
serial number.
auparse_get_time - retrieve time stamp of current record
auparse_get_serial - retrieve serial number of current record
auparse_first_record - set iterator to first record in current event
auparse_next_record - traverse to next record in event. This allows
access to
the event type
auparse_get_type - retrieve type of current record
auparse_first_field - set field pointer to first in current record
auparse_next_field - traverse the fields in a record
auparse_find_field() - find a given field in a event or record
auparse_find_field_next() - find the next occurance of that field inthe
same
record
auparse_get_field_str - return current field value as a string
auparse_get_field_int - return current field value as an int
auparse_interpret_field - interpret the current field as a string
auparse_destroy - free all data structures and close file descriptors
This would allow the following kind of programming:
auparse_init
ausearch_set_param
while ausearch_next_event
if auparse_find_field
auparse_interpret_field
print out
...
auparse_destroy
This is essentially how ausearch works.
The data structures would be hidden from the external application.
Access to
fields is a name/value style. You access the fields through functions
that
either return str pointer or ints.
Would something like this meet everyone's needs?
-Steve
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
11:08 a.m.
On Thursday 09 March 2006 12:03, Debora Velarde wrote:
If I want to match on two params (say syscall name and group id)
would I
call ausearch_set_param twice or pass ausearch_set_param all my parameters
in one call? Can you post how you imagine the call to look like?
Yes, you would call it twice. I would expect it to take 2 params: name &
value.
So you would likely do:
ausearch_set_param("syscall", "open");
ausearch_set_param("gid", "500");
-Steve
11:31 a.m.
linux-audit-bounces(a)redhat.com wrote on 03/09/2006 11:08:05 AM:
On Thursday 09 March 2006 12:03, Debora Velarde wrote:
> If I want to match on two params (say syscall name and group id) would
I
> call ausearch_set_param twice or pass ausearch_set_param all my
parameters
> in one call? Can you post how you imagine the call to look
like?
Yes, you would call it twice. I would expect it to take 2 params: name &
value.
So you would likely do:
ausearch_set_param("syscall", "open");
ausearch_set_param("gid", "500");
Since you are eventually going after Python support, it would be awesome
if (in Pyhton) you could supply a list of pairs, since making multiple
calls is not very friendly.
-Steve
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
1:05 p.m.
Steve Grubb <sgrubb(a)redhat.com> wrote on 03/09/2006 12:55:59 PM:
On Thursday 09 March 2006 12:31, Michael C Thompson wrote:
> since making multiple calls is not very friendly.
Multiple calls are any language friendly. Its just tedious.
Yes, I did not mean that it was unacceptable in terms of the language's
syntax and semantics, but tedious from a coder's viewpoint. However, I
have no suggestion for an alternative that would be better, so I guess
tedious works :)
-Steve
2:24 p.m.
Michael C Thompson <mcthomps(a)us.ibm.com> writes:
Since you are eventually going after Python support, it would be
awesome
if (in Pyhton) you could supply a list of pairs, since making multiple
calls is not very friendly.
I would like speak in support of the simple, thin interface style
Steve promotes, and suggest that a Python library built on top of the
thin layer is the appropriate place to provide functions that
translate lists of pairs into multiple calls to functions defined on
the language boundary.
John
3:12 p.m.
linux-audit-bounces(a)redhat.com wrote on 03/09/2006 02:24:15 PM:
Michael C Thompson <mcthomps(a)us.ibm.com> writes:
> Since you are eventually going after Python support, it would be
awesome
> if (in Pyhton) you could supply a list of pairs, since making
multiple
> calls is not very friendly.
I would like speak in support of the simple, thin interface style
Steve promotes, and suggest that a Python library built on top of the
thin layer is the appropriate place to provide functions that
translate lists of pairs into multiple calls to functions defined on
the language boundary.
Such was my suggestion. Sorry if I failed to make that clear.
Mike
John
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
9:38 p.m.
On Thu, Mar 09, 2006 at 11:31:57AM -0600, Michael C Thompson wrote:
linux-audit-bounces(a)redhat.com wrote on 03/09/2006 11:08:05 AM:
> So you would likely do:
> ausearch_set_param("syscall", "open");
> ausearch_set_param("gid", "500");
Since you are eventually going after Python support, it would be awesome
if (in Pyhton) you could supply a list of pairs, since making multiple
calls is not very friendly.
The Python map() builtin can already do that for you:
map(set_param, ("syscall", "gid", "auid"),
("open", 500, 501))
or, if you prefer the list of pairs instead of separate lists:
map(set_param, *zip(("syscall", "open"), ("gid", 500),
("auid", 501)))
which could of course be wrapped into a set_params() helper function.
As a side note, the high-level interface should be smart enough to take
integers (or general printable objects) instead of insisting on strings.
This wouldn't need to be visible to the low-level interface.
-Klaus
12:49 p.m.
Not sure if ausearch supports this now, but I'm thinking of two use cases:
1. If I want to find all records where the auid is NOT 500
2. If I want to find all records where the gid is greater than 500.
Could we then do:
ausearch_set_param("auid", "!=", "500");
ausearch_set_param("gid", ">", "500");
Steve Grubb <sgrubb(a)redhat.com> wrote on 03/09/2006 09:08:05 AM:
On Thursday 09 March 2006 12:03, Debora Velarde wrote:
> If I want to match on two params (say syscall name and group id) would
I
> call ausearch_set_param twice or pass ausearch_set_param all my
parameters
> in one call? Can you post how you imagine the call to look
like?
Yes, you would call it twice. I would expect it to take 2 params: name &
value.
So you would likely do:
ausearch_set_param("syscall", "open");
ausearch_set_param("gid", "500");
-Steve
1 p.m.
On Thursday 09 March 2006 13:49, Debora Velarde wrote:
Not sure if ausearch supports this now, but I'm thinking of two
use cases:
1. If I want to find all records where the auid is NOT 500
2. If I want to find all records where the gid is greater than 500.
Could we then do:
ausearch_set_param("auid", "!=", "500");
ausearch_set_param("gid", ">", "500");
Sure, sounds fine to me since we are redefining the search engine.
-Steve
1:10 p.m.
I would also vote that we use the field names like they are passed in
auditctl, as opposed to the options passed into ausearch.
So we would use
ausearch_set_param("uid", "!=", "500");
rather than
ausearch_set_param("ui", "!=500");
Steve Grubb <sgrubb(a)redhat.com> wrote on 03/09/2006 11:00:05 AM:
On Thursday 09 March 2006 13:49, Debora Velarde wrote:
> Not sure if ausearch supports this now, but I'm thinking of two use
cases:
> 1. If I want to find all records where the auid is NOT 500
> 2. If I want to find all records where the gid is greater than 500.
>
> Could we then do:
> ausearch_set_param("auid", "!=", "500");
> ausearch_set_param("gid", ">", "500");
Sure, sounds fine to me since we are redefining the search engine.
-Steve
11:05 a.m.
On Thursday 09 March 2006 11:06, Steve Grubb wrote:
> I'll take a hack at proposing an API and send it in a little
while.
The audit library parser could have the following functions:
I think we are in basic agreement...so this is another round of updates with
more details fleshed out. (A CDR if you will.)
Audit Event Parsing Library
====================
All functions that begin with ausearch are related to searching for a subset
of events based on certain criteria. All functions that begin with auparse
are used to access events, records, and fields sequentially and without
regard to any search options that may be in effect. All functions return 1 on
success and 0 on failure unless otherwise noted. Where the return type is a
char, NULL will indicate failure. The data structures would be hidden from
the external application. Access to fields is a name/value style. You access
the fields through functions that either return string pointer or ints.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_BUFFER } source_t;
int auparse_init(source_t source, void *b) - allow init of library. Set data
source: logs, file, buffer. The pointer, b, is used to set the file name or
pass the buff when those types are given.
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD, AUSEARCH_STOP_FIELD
} stop_t;
int ausearch_set_param(const char *field, const char *op, const char *value,
stop_t where) - set search options. The field would be the left hand side of
the audit name value pairs. The op would be how to match, =,!=,>,<, <=, >=.
The value would be the right hand side of the audit field name value pairs.
The where parameter tells the search library where to place the internal
cursor when a match is found. It could be on first field of first record,
first field of record containing the match, or the field that matches.
int ausearch_next_event(void) - traverse to the next event that yields a match
based on the given search criteria.
int auparse_next_event(void) - traverse to next event. This allows access to
time and serial number.
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
} event_t;
event_t auparse_get_timestamp(void) - retrieve time stamp of current record
time_t auparse_get_time(void) - retrieve time in seconds of current record
time_t auparse_get_milli(void) - retrieve milliseconds time of current record
unsigned long auparse_get_serial(void) - retrieve serial number of current
record
int auparse_first_record(void) - set iterator to first record in current event
int auparse_next_record(void) - traverse to next record in event. this allows
access to the event type
int auparse_get_type(void) - retrieve type of current record
int auparse_first_field(void) - set field pointer to first in current record
int auparse_next_field(void) - traverse the fields in a record
const char *auparse_find_field(const char *name) - find a given field in a
event or record. Name is the left hand side of the name/value pair. Returns
pointer to the value as ascii text.
const char *auparse_find_field_next(void ) - find the next occurance of that
field in the same record. Returns pointer to the value as ascii text.
const char *auparse_get_field_str(void) - return current field value as a
string
int auparse_get_field_int(void) - return current field value as an int
const char *auparse_interpret_field(void) - interpret the current field
int auparse_destroy(void) - free all data structures and close file
descriptors
This would allow the following kind of programming:
int main(void)
{
if (!auparse_init(AUSOURCE_LOGS, NULL))
return 1;
if (!ausearch_set_param("auid", "=", "500",
AUSEARCH_STOP_RECORD))
return 1;
while (ausearch_next_event()) {
if (auparse_find_field("auid"))
printf("auid=%s\n", auparse_interpret_field());
}
return 0;
}
11:50 a.m.
Looks good to me. I only have simple editorial comments.
Steve Grubb <sgrubb(a)redhat.com> writes:
success and 0 on failure unless otherwise noted. Where the return
type is a char, NULL will indicate failure.
type is a char pointer, NULL ...
the fields through functions that either return string pointer or
ints.
the fields through functions that either return a pointer to an
immutable, zero-terminated array of ASCII characters or integral
values.
John
1:02 p.m.
On Fri, Mar 10, 2006 at 12:05:27PM -0500, Steve Grubb wrote:
This would allow the following kind of programming:
if (!auparse_init(AUSOURCE_LOGS, NULL))
if (!ausearch_set_param("auid", "=", "500",
AUSEARCH_STOP_RECORD))
while (ausearch_next_event()) {
if (auparse_find_field("auid"))
With this kind of interface, the library will need to keep state
internally, which seems to make it impossible to have multiple scanners
active from the same application. An example for that would be a GUI
application for examining audit logs which may very well want to have
multiple search windows open at once.
How about an extra parameter to make the state management explicit,
similar to the stdio.h opaque FILE* type, to allow fully reentrant
operation?
Something like this:
auparse_state_t *au;
if (! (au = auparse_init(AUSOURCE_LOGS, NULL)))
if (!ausearch_set_param(au, "auid", "=", "500",
AUSEARCH_STOP_RECORD))
while (ausearch_next_event(au, )) {
if (auparse_find_field(au, "auid"))
-Klaus
1:15 p.m.
On Friday 10 March 2006 14:02, Klaus Weidner wrote:
How about an extra parameter to make the state management explicit,
similar to the stdio.h opaque FILE* type, to allow fully reentrant
operation?
Good point. I'll update the spec. I'll wait a little while longer in case
there are other comments before posting an updated spec.
My goal is to finalize the spec today and start implementation next week.
-Steve
1:25 p.m.
Steve,
If not a bother would you mind listing the fields in the record or point
me to a reference of what they are on your next spec?
I was thinking about the kinds of audit parsing we do and I need the
reference to the data.
Thanks,
LCB.
On Fri, 2006-03-10 at 14:15 -0500, Steve Grubb wrote:
On Friday 10 March 2006 14:02, Klaus Weidner wrote:
> How about an extra parameter to make the state management explicit,
> similar to the stdio.h opaque FILE* type, to allow fully reentrant
> operation?
Good point. I'll update the spec. I'll wait a little while longer in case
there are other comments before posting an updated spec.
My goal is to finalize the spec today and start implementation next week.
-Steve
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
--
LC Bruzenak
lenny(a)bruzenak.com
1:32 p.m.
On Friday 10 March 2006 14:25, LC Bruzenak wrote:
If not a bother would you mind listing the fields in the record or
point
me to a reference of what they are on your next spec?
We'll make that later in the project. I would need to spend some time going
over every single message. Amazingly, you can write a parser without knowing
what all is there since it all follows a well defined pattern.
-Steve
1:42 p.m.
On Fri, 2006-03-10 at 14:32 -0500, Steve Grubb wrote:
On Friday 10 March 2006 14:25, LC Bruzenak wrote:
> If not a bother would you mind listing the fields in the record or point
> me to a reference of what they are on your next spec?
We'll make that later in the project. I would need to spend some time going
over every single message. Amazingly, you can write a parser without knowing
what all is there since it all follows a well defined pattern.
-Steve
OK; that's true. The name/value pairs can be tweaked later.
I just got lost on the following IF descriptions:
...
const char *auparse_find_field(const char *name) - find a given field in
a event or record. Name is the left hand side of the name/value pair.
Returns pointer to the value as ascii text.
const char *auparse_find_field_next(void ) - find the next occurance of
that field in the same record. Returns pointer to the value as ascii
text.
...
I didn't understand how in the first case you had 1 named field and in
the next there was more than one occurrence of that field. Seems to me
that (unless I misunderstand) you may in the first case think you know
the field name and stop processing that record ... but the second one
implies there are other fields by the same name.
That to me means that the field names are not unique; hence my question.
LCB.
--
LC Bruzenak
lenny(a)bruzenak.com
1:50 p.m.
On Friday 10 March 2006 14:42, LC Bruzenak wrote:
I didn't understand how in the first case you had 1 named field
and in
the next there was more than one occurrence of that field.
It would be stored internally for reference. Think strtok.
Seems to me that (unless I misunderstand) you may in the first case
think
you know the field name and stop processing that record ... but the second
one implies there are other fields by the same name.
Sort of...there are multiple fields with the same name in various records that
make up an event. The idea is that you can pass the name one time and then
its stored internally until another call replaces it.
That to me means that the field names are not unique; hence my
question.
This is true. An example is uid. I think there are some places where the
kernel inserts uid and userspace inserts uid to the same event.
-Steve
1:53 p.m.
On Fri, Mar 10, 2006 at 01:42:00PM -0600, LC Bruzenak wrote:
That to me means that the field names are not unique; hence my
question.
There's two separate issues here:
- audit records that contain the same field name twice for different
purposes in a single record. I think this happens in a couple of places
where uid or something like that is re-used. My preference would be to
consider this a bug in the audit generation that needs fixing, instead
of having the parser handle it. (As a side note, any remaining tag names
containing spaces should also be fixed...)
- multiple related audit records for a single event that contain several
instances of the same tag, for example a syscall such as rename() that
generates multiple path tags for source and destination. I'm not sure
how those get handled, is that what this is intended for?
Does the auparse library handle merging of related records for single
events, or is that left for higher level code?
-Klaus
2:32 p.m.
On Fri, 2006-03-10 at 13:53 -0600, Klaus Weidner wrote:
On Fri, Mar 10, 2006 at 01:42:00PM -0600, LC Bruzenak wrote:
> That to me means that the field names are not unique; hence my question.
There's two separate issues here:
- audit records that contain the same field name twice for different
purposes in a single record. I think this happens in a couple of places
where uid or something like that is re-used. My preference would be to
consider this a bug in the audit generation that needs fixing, instead
of having the parser handle it. (As a side note, any remaining tag names
containing spaces should also be fixed...)
Yes, and the audit scanner(s) would have to handle/accommodate this
also.
In this scheme is there a distinction between name/value pairs supplied
by the kernel vice those added by userspace or audit-aware applications?
LCB.
--
LC Bruzenak
lenny(a)bruzenak.com
2:38 p.m.
On Friday 10 March 2006 14:53, Klaus Weidner wrote:
- audit records that contain the same field name twice for different
purposes in a single record. I think this happens in a couple of places
where uid or something like that is re-used. My preference would be to
consider this a bug in the audit generation that needs fixing, instead
of having the parser handle it.
I'm ambivalent about this. The current behavior is to favor the uid in the
message instead of the kernel supplied one. For example, root change bob's
account password. search for uid=bob will hit on the password change event
since bob's account is what's being manipulated.
(As a side note, any remaining tag names containing spaces should
also be
fixed...)
Sure...point them out.
- multiple related audit records for a single event that contain
several
instances of the same tag, for example a syscall such as rename() that
generates multiple path tags for source and destination. I'm not sure
how those get handled, is that what this is intended for?
Could be. Or maybe you just want a certain field and that's all. In this case,
you set the search item the first time and then step through the records with
auparse_find_field_next.
auparse_find_field("auid");
while (auparse_next_event()) {
while (auparse_next_record()) {
while (auparse_find_field_next()) {
do-something
}
}
}
BTW, without this function, you wouldn't have a way to verify that one and
only one field type is emitted for each record if we decide to cleanup the
messages.
Does the auparse library handle merging of related records for
single
events, or is that left for higher level code?
The library would group the records into events and let you step through them.
while (auparse_next_event()) {
while (auparse_next_record()) {
while (auparse_next_field()) {
do-something
}
}
}
-Steve
2:42 p.m.
On Friday 10 March 2006 15:38, Steve Grubb wrote:
For example, root change bob's
account password. search for uid=bob will hit on the password change event
since bob's account is what's being manipulated.
Actually, this is bad example since I don't think it happens for this message.
There's only 1 or 2 messages that do this and I don't remember what they are
other than user space. But we do favor the uid later in the message under the
current scheme.
-Steve
8:57 a.m.
Klaus Weidner wrote:
On Fri, Mar 10, 2006 at 01:42:00PM -0600, LC Bruzenak wrote:
>That to me means that the field names are not unique; hence my question.
There's two separate issues here:
- audit records that contain the same field name twice for different
purposes in a single record. I think this happens in a couple of places
where uid or something like that is re-used. My preference would be to
consider this a bug in the audit generation that needs fixing, instead
of having the parser handle it. (As a side note, any remaining tag names
containing spaces should also be fixed...)
On the Side note issue, I am all for that, using a "space" when
"_"
should be just makes for alot of unnecessary parsing exceptions to skip
those lonely words.
Also, many audit records have what seems to me to be random symbols (ex.
, : ( ' ). If we get rid of those .. that would be great.
If that is something we want, I can create a patch to fix these
oddities. I believe those messages come from kernel.. right?
Are there any that come from audit userspace?
- Loulwa
9:18 a.m.
On Monday 13 March 2006 09:57, Loulwa Salem wrote:
On the Side note issue, I am all for that, using a "space"
when "_"
should be just makes for alot of unnecessary parsing exceptions to skip
those lonely words.
That would actually slow down parsing since I would now have to do lots of
exception processing. It has to be high performance and adding spaces just
makes that harder to achieve.
Also, many audit records have what seems to me to be random symbols
(ex.
, : ( ' ). If we get rid of those .. that would be great.
The are separators for different kinds of information. Don't worry about any
of these details. auparse library should make it such that you don't worry
about the underlying details.
-Steve
9:28 a.m.
Steve Grubb wrote:
>On the Side note issue, I am all for that, using a
"space" when "_"
>should be just makes for alot of unnecessary parsing exceptions to skip
>those lonely words.
That would actually slow down parsing since I would now have to do lots of
exception processing. It has to be high performance and adding spaces just
makes that harder to achieve.
exactly my point ... I have to deal with the same
exceptions for our
parser. Best thing and what makes the most sense is to use "audit_uid"
for example instead of "audit uid" and so on for all those two word fields.
>Also, many audit records have what seems to me to be random
symbols (ex.
> , : ( ' ). If we get rid of those .. that would be great.
The are separators for different kinds of information. Don't worry about any
of these details. auparse library should make it such that you don't worry
about the underlying details.
Well .. our parser right now just skips those symbols
when it encounters
them, but I wanted to bring it up to better understand what purpose they
serve. If they are necessary, then I guess they stay.
-loulwa
10:54 a.m.
On Friday 10 March 2006 14:25, LC Bruzenak wrote:
If not a bother would you mind listing the fields in the record or
point
me to a reference of what they are on your next spec?
I just updated the audit library parsing spec to include all this information.
I don't mention what records they are in. Instead it lists what all the field
definitions are.
You can find it here:
http://people.redhat.com/sgrubb/audit/audit-parse.txt
If anyone sees something missing from the list or changes, please let me know.
-Steve
6:16 a.m.
Steve Grubb <sgrubb(a)redhat.com> writes:
I just updated the audit library parsing spec to include all this
information. I don't mention what records they are in. Instead it
lists what all the field definitions are.
Steve,
This is very helpful information. I expect I will be acquiring field
values using auparse_interpret_field, not auparse_get_field_str, so I
would really like to see the field definitions augmented with a
description of their results when the field is interpreted. When
there is a difference, perhaps you could describe it within say
parentheses.
I'd like to engage you in a thought experiment. Suppose the task is
to generate strace like output from an audit log. To make the
experiment concrete, let's suppose each event is given to you as a
sequence of Python dictionaries, and each dictionary contains the
content of a single audit record. As near as I can tell, here is the
beginnings of an algorithm that can do the job.
Let seq be a sequence of dictionaries representing an audit event.
1. If the sequence contains no dictionary that maps "type" to
"SYSCALL", process the next event.
2. Set n to be the index in seq of the dictionary that maps "type" to
"SYSCALL".
3. If seq[n]["syscall"] contains a parenthesis, goto step 10.
4. Print seq[n]["syscall"]
5. Let i be seq[n]["items"] or zero if seq[n]["items"] is not
defined.
6. For j from 0 to i-1, print the "name" field from the dictionary
that maps "type" to "PATH", and "item" to j.
7. For j from i+1 by 1, while seq[n]["a" + str(j)] is defined, print
seq[n]["a" + str(j)].
8. Print seq[n]["exit"].
9. Process the next event.
10. If seq[n]["syscall"] does not match "socketcall([^)]+)", goto 16.
11. Print the capture that results from matching "socketcall([^)]+)".
12. If there is a dictionary that maps "type" to "SOCKADDR", print
saddr as the first argument to the system call, otherwise use
the dictionary that maps "type" to "SOCKETCALL", to print the
first argument.
13. Print the remaining arguments using the dictionary that maps
"type" to "SOCKETCALL".
14. Print seq[n]["exit"].
15. Process the next event.
16. I haven't figured out how to handle the ipc system call.
Is this algorithm correct? Perhaps the audit-parse.txt document
should contain a description of the correct algorithm.
John
You can find it here:
http://people.redhat.com/sgrubb/audit/audit-parse.txt
3:33 p.m.
On Friday 10 March 2006 12:05, Steve Grubb wrote:
so this is another round of updates with more details fleshed out.
OK. I think the last round of comments was helpful. I added some language to
define the concept of multiple hosts in one log. This is the final draft
unless there is an omission.
-Steve
Audit Event Parsing Library Specifications
==========================================
Definitions
-----------
An audit event is all records that have the same host, timestamp, and
serial number. Each event on a host has a unique timestamp and serial
number. An event is composed of multiple records which have information about
different aspects of an audit event. Each record is denoted by a type which
indicates what fields will follow. Information in the fields are held by a
name/value pair that contains an '=' between them. Each field is separated
from one another by a space or comma.
Ground Rules
------------
All functions that begin with ausearch are related to searching for a subset
of events based on certain criteria. All functions that begin with auparse
are used to access events, records, and fields sequentially and without
regard to any search options that may be in effect. All functions return 1
on success and 0 on failure unless otherwise noted. Where the return type is
a char pointer, NULL will indicate failure. The data structures would be
hidden from the external application. Access to fields is a name/value style.
You access the fields through functions that either return a pointer to an
immutable, zero-terminated array of ASCII characters or integral values. Every
function (except auparse_init) takes a parameter, au, which is the internal
state information for the current query.
Functions
---------
auparse_state_t - is an opaque data type used for maintaining library state.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_BUFFER } ausource_t;
auparse_state_t *auparse_init(ausource_t source, const void *b) - allow init
of library. Set data source: logs, file, buffer. The pointer 'b' is used to
set the file name or pass the buff when those types are given.
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD,
AUSEARCH_STOP_FIELD } austop_t;
int ausearch_set_param(auparse_state_t *au, const char *field, const char *op,
const char *value, austop_t where) - set search
options. The field would be the left hand side of the audit name/value pairs.
The op would be how to match: =,!=,>,<. The value would be the right hand
side of the audit field name/value pairs. The where parameter tells the
search library where to place the internal cursor when a match is found. It
could be on first field of first record, first field of record containing the
match, or the field that matches.
int ausearch_next_event(auparse_state_t *au) - traverse to the next event that
yields a match based on the given search criteria.
int auparse_next_event(auparse_state_t *au) - traverse to next event. This
allows access to time and serial number.
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
const char *host; // Machine's name
} event_t;
event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time stamp of
current record
time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds of
current record
time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds time of
current record
unsigned long auparse_get_serial(auparse_state_t *au) - retrieve serial number
of current record
const char *auparse_get_host(auparse_state_t *au) - retrieve host name
of current record
int auparse_first_record(auparse_state_t *au) - set iterator to first record
in current event
int auparse_next_record(auparse_state_t *au) - traverse to next record in
event. This allows access to the event type
int auparse_get_type(auparse_state_t *au) - retrieve type of current record
int auparse_first_field(auparse_state_t *au) - set field pointer to first in
current record
int auparse_next_field(auparse_state_t *au) - traverse the fields in a record
const char *auparse_find_field(auparse_state_t *au, const char *name) - find a
given field in a event or record. Name is the left hand side of the name/value
pair. Returns pointer to the value as ascii text.
const char *auparse_find_field_next(auparse_state_t *au) - find the next
occurance of that field in the same record. Returns pointer to the value as
ascii text.
const char *auparse_get_field_str(auparse_state_t *au) - return current field
value as a string
int auparse_get_field_int(auparse_state_t *au) - return current field value
as an int
const char *auparse_interpret_field(auparse_state_t *au) - interpret the
current field
int auparse_destroy(auparse_state_t *au) - free all data structures and close
file descriptors
Code Example
------------
int main(void)
{
auparse_state_t *au = auparse_init(AUSOURCE_LOGS, NULL);
if (au == NULL)
exit(1);
if (!ausearch_set_param(au, "auid", "=", "500",
AUSEARCH_STOP_EVENT))
exit(1);
while (ausearch_next_event(au)) {
if (auparse_find_field(au, "auid")) {
printf("auid=%s\n", auparse_interpret_field(au));
}
}
return 0;
}
4:45 p.m.
A couple questions about how I would access all the info in a record such
as:
type=USER_ACCT msg=audit(1127329315.111:55495): user pid=23769 uid=0
auid=4294967295 msg='PAM accounting: user=laf_a exe="/usr/sbin/sshd"
(hostname=system.ibm.com, addr=127.0.0.1, terminal=ssh result=Success)'
1. In this case would auparse_get_host(auparse_state_t *au) retrieve the
hostname of this record?
2. Will the user have to extract the values of 'user' and 'exe' from the
entire value of 'msg' themselves? Or can the API return the values for
those individually?
linux-audit-bounces(a)redhat.com wrote on 03/10/2006 01:33:12 PM:
On Friday 10 March 2006 12:05, Steve Grubb wrote:
> so this is another round of updates with more details fleshed out.
OK. I think the last round of comments was helpful. I added some
language to
define the concept of multiple hosts in one log. This is the final
draft
unless there is an omission.
-Steve
Audit Event Parsing Library Specifications
==========================================
Definitions
-----------
An audit event is all records that have the same host, timestamp, and
serial number. Each event on a host has a unique timestamp and serial
number. An event is composed of multiple records which have information
about
different aspects of an audit event. Each record is denoted by a type
which
indicates what fields will follow. Information in the fields are held
by
a
name/value pair that contains an '=' between them. Each field
is
separated
from one another by a space or comma.
Ground Rules
------------
All functions that begin with ausearch are related to searching for a
subset
of events based on certain criteria. All functions that begin with
auparse
are used to access events, records, and fields sequentially and
without
regard to any search options that may be in effect. All functions return
1
on success and 0 on failure unless otherwise noted. Where the return
type is
a char pointer, NULL will indicate failure. The data structures would
be
hidden from the external application. Access to fields is a name/value
style.
You access the fields through functions that either return a pointer
to
an
immutable, zero-terminated array of ASCII characters or integral
values.
Every
function (except auparse_init) takes a parameter, au, which is the
internal
state information for the current query.
Functions
---------
auparse_state_t - is an opaque data type used for maintaining library
state.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_BUFFER }
ausource_t;
auparse_state_t *auparse_init(ausource_t source, const void *b) - allow
init
of library. Set data source: logs, file, buffer. The pointer
'b' is used
to
set the file name or pass the buff when those types are given.
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD,
AUSEARCH_STOP_FIELD } austop_t;
int ausearch_set_param(auparse_state_t *au, const char *field, constchar
*op,
const char *value, austop_t where) - set
search
options. The field would be the left hand side of the audit name/value
pairs.
The op would be how to match: =,!=,>,<. The value would be the
right
hand
side of the audit field name/value pairs. The where parameter tells
the
search library where to place the internal cursor when a match is found.
It
could be on first field of first record, first field of record
containing the
match, or the field that matches.
int ausearch_next_event(auparse_state_t *au) - traverse to the next
event that
yields a match based on the given search criteria.
int auparse_next_event(auparse_state_t *au) - traverse to next event.
This
allows access to time and serial number.
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
const char *host; // Machine's name
} event_t;
event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time stamp
of
current record
time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds
of
current record
time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds
time of
current record
unsigned long auparse_get_serial(auparse_state_t *au) - retrieve serial
number
of current record
const char *auparse_get_host(auparse_state_t *au) - retrieve host name
of current record
int auparse_first_record(auparse_state_t *au) - set iterator to first
record
in current event
int auparse_next_record(auparse_state_t *au) - traverse to next record
in
event. This allows access to the event type
int auparse_get_type(auparse_state_t *au) - retrieve type of current
record
int auparse_first_field(auparse_state_t *au) - set field pointer to
first in
current record
int auparse_next_field(auparse_state_t *au) - traverse the fields in a
record
const char *auparse_find_field(auparse_state_t *au, const char *name) -
find a
given field in a event or record. Name is the left hand side of the
name/value
pair. Returns pointer to the value as ascii text.
const char *auparse_find_field_next(auparse_state_t *au) - find the next
occurance of that field in the same record. Returns pointer to the value
as
ascii text.
const char *auparse_get_field_str(auparse_state_t *au) - return current
field
value as a string
int auparse_get_field_int(auparse_state_t *au) - return current field
value
as an int
const char *auparse_interpret_field(auparse_state_t *au) - interpret the
current field
int auparse_destroy(auparse_state_t *au) - free all data structures and
close
file descriptors
Code Example
------------
int main(void)
{
auparse_state_t *au = auparse_init(AUSOURCE_LOGS, NULL);
if (au == NULL)
exit(1);
if (!ausearch_set_param(au, "auid", "=", "500",
AUSEARCH_STOP_EVENT))
exit(1);
while (ausearch_next_event(au)) {
if (auparse_find_field(au, "auid")) {
printf("auid=%s\n", auparse_interpret_field(au));
}
}
return 0;
}
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
5:13 p.m.
On Friday 10 March 2006 17:45, Debora Velarde wrote:
1. In this case would auparse_get_host(auparse_state_t *au) retrieve
the
hostname of this record?
It would retrieve the name of the machine that the audit message came from. In
this format, it would default to the rough equivalent of "uname -n". The
record format will change to accommodate a host field. This is needed so that
a data center can have a central logger that stores everything.
2. Will the user have to extract the values of 'user' and
'exe' from the
entire value of 'msg' themselves?
No. look at the example code. You would do
if (auparse_find_field(au, "user") {
const char *str = auparse_get_field_str(au);
do-whatever(str);
}
Or can the API return the values for those individually?
Yes.
-Steve
5:51 p.m.
> 2. Will the user have to extract the values of 'user'
and 'exe' from
the
> entire value of 'msg' themselves?
No. look at the example code. You would do
if (auparse_find_field(au, "user") {
const char *str = auparse_get_field_str(au);
do-whatever(str);
}
> Or can the API return the values for those individually?
Yes.
-Steve
OK Great. I just wasn't sure since 'user' and 'exe' are
within the 'msg'
string.
9:50 a.m.
On Friday 10 March 2006 17:45, Debora Velarde wrote:
1. In this case would auparse_get_host(auparse_state_t *au) retrieve
the
hostname of this record?
Since this is introducing the notion of multiple machines potentially sharing
the same log...would it be more clear to change the name to prevent
confusion?
Its currently host, but would could make it: server, node, machine, etc.
-Steve
11:45 a.m.
linux-audit-bounces(a)redhat.com wrote on 03/13/2006 09:50:46 AM:
On Friday 10 March 2006 17:45, Debora Velarde wrote:
> 1. In this case would auparse_get_host(auparse_state_t *au) retrieve
the
> hostname of this record?
Since this is introducing the notion of multiple machines potentially
sharing
the same log...would it be more clear to change the name to prevent
confusion?
Its currently host, but would could make it: server, node, machine, etc.
OK, I have a question because I think there are two issues here:
1) Steve has a function, called auparse_get_host, which is paralleled
by other things like get_serial, which implies that every event has an
associated host
2) This particular event has a host field, I do not think that
auparse_get_host(0 and auparse_get_filed(au,"host") are the same
Do I understand this issue correctly? And if this is the case, is your
reply Debbie's first question still true?
-Steve
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
12:29 p.m.
On Monday 13 March 2006 12:45, Michael C Thompson wrote:
Do I understand this issue correctly?
I think so.
And if this is the case, is your reply Debbie's first question
still true?
My answer is still true. I was just wondering if the name is going to cause
other people to mistunderstand its meaning.
-Steve
8:35 a.m.
linux-audit-bounces(a)redhat.com wrote on 03/10/2006 03:33:12 PM:
On Friday 10 March 2006 12:05, Steve Grubb wrote:
> so this is another round of updates with more details fleshed out.
OK. I think the last round of comments was helpful. I added some
language to
define the concept of multiple hosts in one log. This is the final
draft
unless there is an omission.
-Steve
Audit Event Parsing Library Specifications
==========================================
Definitions
-----------
An audit event is all records that have the same host, timestamp, and
serial number. Each event on a host has a unique timestamp and serial
number. An event is composed of multiple records which have information
about
different aspects of an audit event. Each record is denoted by a type
which
indicates what fields will follow. Information in the fields are held
by
a
name/value pair that contains an '=' between them. Each field
is
separated
from one another by a space or comma.
I know this is not likely to change, but is there a reason why we don't
have a common delimiter? I understand that going through the code and
modifying this would be a large undertaking, and I'm not even saying I
will do it (although I believe Loulwa was consider doing at one point, but
don't hold me to that). Removing inconstancies in the records, such as
using '_' in place of spaces in field names and removing these commas,
would seem like a good thing to do, but before anyone does that work, will
making that change be acceptable from a code point of view? (Discount any
work which might be needed to update parsing code from ausearch etc,
although there would be a fair amount to change I imagine).
Ground Rules
------------
All functions that begin with ausearch are related to searching for a
subset
of events based on certain criteria. All functions that begin with
auparse
are used to access events, records, and fields sequentially and
without
regard to any search options that may be in effect. All functions return
1
on success and 0 on failure unless otherwise noted. Where the return
type is
a char pointer, NULL will indicate failure. The data structures would
be
Typo: The data structures [would -> will] be
hidden from the external application. Access to fields is a
name/value
style.
You access the fields through functions that either return a pointer
to
an
immutable, zero-terminated array of ASCII characters or integral
values.
Every
function (except auparse_init) takes a parameter, au, which is the
internal
state information for the current query.
Although it is clear from the code example (and I am sure this is not
going to be the final version of the document, it would probably be nice
to mention that auparse will, when not using auparse_next_event, will be
parsing the record that was found (and is current being pointed to) by
ausearch_* functions, for example. Your comment in the above, "without
regard to any search options that may be in effect", is somewhat
befuddling on this point. A clear statement such as "auparse_* functions
will parse the current event found either through ausearch_* functions or
auparse_* functions." I am not a writer though, so don't use my suggestion
verbatim.
Functions
---------
auparse_state_t - is an opaque data type used for maintaining library
state.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_BUFFER }
ausource_t;
auparse_state_t *auparse_init(ausource_t source, const void *b) - allow
init
of library. Set data source: logs, file, buffer. The pointer
'b' is used
to
set the file name or pass the buff when those types are given.
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD,
AUSEARCH_STOP_FIELD } austop_t;
int ausearch_set_param(auparse_state_t *au, const char *field, constchar
*op,
const char *value, austop_t where) - set
search
options. The field would be the left hand side of the audit name/value
pairs.
The op would be how to match: =,!=,>,<. The value would be the
right
hand
side of the audit field name/value pairs. The where parameter tells
the
search library where to place the internal cursor when a match is found.
It
could be on first field of first record, first field of record
containing the
match, or the field that matches.
int ausearch_next_event(auparse_state_t *au) - traverse to the next
event that
yields a match based on the given search criteria.
int auparse_next_event(auparse_state_t *au) - traverse to next event.
This
allows access to time and serial number.
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
const char *host; // Machine's name
} event_t;
event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time stamp
of
current record
time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds
of
current record
time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds
time of
current record
unsigned long auparse_get_serial(auparse_state_t *au) - retrieve serial
number
of current record
const char *auparse_get_host(auparse_state_t *au) - retrieve host name
of current record
int auparse_first_record(auparse_state_t *au) - set iterator to first
record
in current event
int auparse_next_record(auparse_state_t *au) - traverse to next record
in
event. This allows access to the event type
int auparse_get_type(auparse_state_t *au) - retrieve type of current
record
int auparse_first_field(auparse_state_t *au) - set field pointer to
first in
current record
int auparse_next_field(auparse_state_t *au) - traverse the fields in a
record
const char *auparse_find_field(auparse_state_t *au, const char *name) -
find a
given field in a event or record. Name is the left hand side of the
name/value
pair. Returns pointer to the value as ascii text.
const char *auparse_find_field_next(auparse_state_t *au) - find the next
occurance of that field in the same record. Returns pointer to the value
as
ascii text.
const char *auparse_get_field_str(auparse_state_t *au) - return current
field
value as a string
int auparse_get_field_int(auparse_state_t *au) - return current field
value
as an int
const char *auparse_interpret_field(auparse_state_t *au) - interpret the
current field
int auparse_destroy(auparse_state_t *au) - free all data structures and
close
file descriptors
Code Example
------------
int main(void)
{
auparse_state_t *au = auparse_init(AUSOURCE_LOGS, NULL);
if (au == NULL)
exit(1);
if (!ausearch_set_param(au, "auid", "=", "500",
AUSEARCH_STOP_EVENT))
exit(1);
while (ausearch_next_event(au)) {
if (auparse_find_field(au, "auid")) {
printf("auid=%s\n", auparse_interpret_field(au));
}
}
return 0;
}
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
9:14 a.m.
On Monday 13 March 2006 09:35, Michael C Thompson wrote:
I know this is not likely to change, but is there a reason why we
don't
have a common delimiter?
Personally, I think its a bad idea since it makes the text "darker". Spaces
make it lighter and more pleasing to the eye. Its too later to make these
kind of changes.
Removing inconstancies in the records, such as using '_' in
place of spaces
in field names and removing these commas, would seem like a good thing to
do, but before anyone does that work, will making that change be acceptable
from a code point of view?
I would like to fix inconsistencies, but I don't consider '_' to be an
inconsistency. I would be hesitant to make any other changes besides
normalizing records. Its unnecessary and unwanted.
Although it is clear from the code example (and I am sure this is
not
going to be the final version of the document, it would probably be nice
to mention that auparse will, when not using auparse_next_event, will be
parsing the record that was found (and is current being pointed to) by
ausearch_* functions, for example.
But this is not true.
Your comment in the above, "without regard to any search options
that may be
in effect", is somewhat befuddling on this point.
OK, you put in a search for uid=500. ausearch will find events that have a
match. You should call ausearch functions to walk these events. So, for
example, the cursor goes to event 150 on first call, then event 200 next
call. If you auparse, then it will go to the next event which could be 201.
This is all common database techniques. One is based off an index and the
other is sequential access
-Steve
10:06 a.m.
Steve Grubb <sgrubb(a)redhat.com> wrote on 03/13/2006 09:14:22 AM:
On Monday 13 March 2006 09:35, Michael C Thompson wrote:
> I know this is not likely to change, but is there a reason why we
don't
> have a common delimiter?
Personally, I think its a bad idea since it makes the text "darker".
Spaces
make it lighter and more pleasing to the eye. Its too later to make
these
kind of changes.
Wow, what I wrote is not what I mean. I should say that I am _not_ in
favor of commas, I was trying to point out that spaces between fields (and
only spaces), should be the way to go since its the most common form in
our records and should therefore be made the only form for consistancy -
not to mention all of the reasons you just listed. Sorry for not writing
what I meant.
> Removing inconstancies in the records, such as using '_' in place of
spaces
> in field names and removing these commas, would seem like a good
thing
to
> do, but before anyone does that work, will making that change be
acceptable
> from a code point of view?
I would like to fix inconsistencies, but I don't consider '_' to be an
inconsistency. I would be hesitant to make any other changes besides
normalizing records. Its unnecessary and unwanted.
Again, I was not clear, although when I wrote this, I thought I was. I was
saying I am in favor of chaning field names, such as "login uid", and
replacing them with "login_uid". Amazing how much meaning can be lost
between the reader and the writer. Normalizing records is good!
> Although it is clear from the code example (and I am sure this is not
> going to be the final version of the document, it would probably be
nice
> to mention that auparse will, when not using auparse_next_event,
will
be
> parsing the record that was found (and is current being pointed
to) by
> ausearch_* functions, for example.
But this is not true.
It isn't? From the code example that I read, although it does not have any
documentation comments with it, leads me to believe my understanding to be
correct. btw, you forgot to free your memory in that example! See the next
comment for more.
> Your comment in the above, "without regard to any search options that
may be
> in effect", is somewhat befuddling on this point.
OK, you put in a search for uid=500. ausearch will find events that have
a
match. You should call ausearch functions to walk these events. So,
for
example, the cursor goes to event 150 on first call, then event 200 next
call. If you auparse, then it will go to the next event which could
be
201.
This is all common database techniques. One is based off an index and
the
other is sequential access.
OK, we've got a serious breakdown in communication here. Yes, I understand
that ausearch_next_event will get you to the next event that matches your
criteria. And I also understand that auPARSE_next_event will get you to
the next event that falls in chronologial order (in terms of serial number
or time stamp, whatever) and that is _not_ subject to any specified search
criteria. However, my sole point was it would be good to state clearly
that all other auparse_* functions, which concern themselves with parsing
of records and fields, will do so based on the current event they are at,
whether you got to it through an ausearch_next_event call or an
auparse_next_event call.
Mike
10:14 a.m.
I apologize for jumping into this thread so late, it has been real busy
around Tresys lately :) I think everything looks good but I have a question
about how you imagine regular expression matching to work. For example if I
want to match the "exe" field in avc messages with at regular expression,
what will be the best way given this API?
Does it become something like:
ausearch_set_param("exe", "regex", "/sbin(/.*)")
Kevin Carr
Tresys Technology
410.290.1411 x137
On Friday 10 March 2006 12:05, Steve Grubb wrote:
> so this is another round of updates with more details fleshed out.
OK. I think the last round of comments was helpful. I added some language
to
define the concept of multiple hosts in one log. This is the final draft
unless there is an omission.
-Steve
Audit Event Parsing Library Specifications
==========================================
Definitions
-----------
An audit event is all records that have the same host, timestamp, and
serial number. Each event on a host has a unique timestamp and serial
number. An event is composed of multiple records which have information
about
different aspects of an audit event. Each record is denoted by a type
which
indicates what fields will follow. Information in the fields are held by a
name/value pair that contains an '=' between them. Each field is separated
from one another by a space or comma.
Ground Rules
------------
All functions that begin with ausearch are related to searching for a
subset
of events based on certain criteria. All functions that begin with auparse
are used to access events, records, and fields sequentially and without
regard to any search options that may be in effect. All functions return 1
on success and 0 on failure unless otherwise noted. Where the return type
is
a char pointer, NULL will indicate failure. The data structures would be
hidden from the external application. Access to fields is a name/value
style.
You access the fields through functions that either return a pointer to an
immutable, zero-terminated array of ASCII characters or integral values.
Every
function (except auparse_init) takes a parameter, au, which is the
internal
state information for the current query.
Functions
---------
auparse_state_t - is an opaque data type used for maintaining library
state.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_BUFFER } ausource_t;
auparse_state_t *auparse_init(ausource_t source, const void *b) - allow
init
of library. Set data source: logs, file, buffer. The pointer 'b' is used
to
set the file name or pass the buff when those types are given.
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD,
AUSEARCH_STOP_FIELD } austop_t;
int ausearch_set_param(auparse_state_t *au, const char *field, const char
*op,
const char *value, austop_t where) - set search
options. The field would be the left hand side of the audit name/value
pairs.
The op would be how to match: =,!=,>,<. The value would be the right hand
side of the audit field name/value pairs. The where parameter tells the
search library where to place the internal cursor when a match is found.
It
could be on first field of first record, first field of record containing
the
match, or the field that matches.
int ausearch_next_event(auparse_state_t *au) - traverse to the next event
that
yields a match based on the given search criteria.
int auparse_next_event(auparse_state_t *au) - traverse to next event. This
allows access to time and serial number.
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
const char *host; // Machine's name
} event_t;
event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time stamp
of
current record
time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds of
current record
time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds time
of
current record
unsigned long auparse_get_serial(auparse_state_t *au) - retrieve serial
number
of current record
const char *auparse_get_host(auparse_state_t *au) - retrieve host name
of current record
int auparse_first_record(auparse_state_t *au) - set iterator to first
record
in current event
int auparse_next_record(auparse_state_t *au) - traverse to next record in
event. This allows access to the event type
int auparse_get_type(auparse_state_t *au) - retrieve type of current
record
int auparse_first_field(auparse_state_t *au) - set field pointer to first
in
current record
int auparse_next_field(auparse_state_t *au) - traverse the fields in a
record
const char *auparse_find_field(auparse_state_t *au, const char *name) -
find a
given field in a event or record. Name is the left hand side of the
name/value
pair. Returns pointer to the value as ascii text.
const char *auparse_find_field_next(auparse_state_t *au) - find the next
occurance of that field in the same record. Returns pointer to the value
as
ascii text.
const char *auparse_get_field_str(auparse_state_t *au) - return current
field
value as a string
int auparse_get_field_int(auparse_state_t *au) - return current field
value
as an int
const char *auparse_interpret_field(auparse_state_t *au) - interpret the
current field
int auparse_destroy(auparse_state_t *au) - free all data structures and
close
file descriptors
Code Example
------------
int main(void)
{
auparse_state_t *au = auparse_init(AUSOURCE_LOGS, NULL);
if (au == NULL)
exit(1);
if (!ausearch_set_param(au, "auid", "=", "500",
AUSEARCH_STOP_EVENT))
exit(1);
while (ausearch_next_event(au)) {
if (auparse_find_field(au, "auid")) {
printf("auid=%s\n", auparse_interpret_field(au));
}
}
return 0;
}
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
10:44 a.m.
On Monday 13 March 2006 11:14, Kevin Carr wrote:
I have a question about how you imagine regular expression matching
to
work. For example if I want to match the "exe" field in avc messages with
at regular expression, what will be the best way given this API?
That's a good question. Right now, ausearch has 2 kinds of matching, substring
and whole string. Which match to use is governed by the -w parameter to
ausearch. The default being substring.
Does it become something like:
ausearch_set_param("exe", "regex", "/sbin(/.*)")
I think we need to add another operator to correctly represent ausearch's
capabilities:
ausearch_set_param("exe", "~", "/sbin");
For example, the above would do substring matches. Any exe field that
has /sbin, would match. So both /sbin/fdisk and /usr/sbin/nstat would match.
ausearch_set_param("exe", "=", "/sbin");
Would likely draw no matches since a directory isn't an executable.
I have stayed away from regex matching because performance will be bad.
However, I think we can throw that in at this point and just warn people that
its likely to be slow.
I'll update the spec to include a section about operators & matching and
repost it to this thread.
-Steve
10:42 a.m.
Steve Grubb <sgrubb(a)redhat.com> writes:
Each record is denoted by a type which indicates what fields will
follow. Information in the fields are held by a name/value pair that
contains an '=' between them. Each field is separated from one
another by a space or comma.
Please do not separate fields with commas. The length of each line is
way too long as it is. Furthermore, when ausearch interprets numeric
entities into text, there is a simple, lex-based program can
format the output into XML with the following DTD:
<!ELEMENT au (seq)*>
<!ELEMENT seq (tab)+>
<!ELEMENT tab (ent)+>
<!ELEMENT ent EMPTY>
<!ATTLIST ent
key CDATA #REQUIRED
val CDATA #IMPLIED>
The output can then be consumed with another, very simple Python
program:
--------------------- consume.py -------------------------
import sys, xml.sax, xml.sax.handler
def main():
if len(sys.argv) != 2:
print "Usage: " + sys.argv[0] + " FILE"
else:
xml.sax.parse(sys.argv[1], AuditHandler())
class AuditHandler(xml.sax.handler.ContentHandler):
seq = None
tab = None
def startElement(self, name, attrs):
if name == 'seq':
self.seq = []
elif name == 'tab':
self.tab = {}
elif name == 'ent' and attrs.has_key("key") and
attrs.has_key("val"):
self.tab[attrs.getValue("key")] = attrs.getValue("val")
def endElement(self, name):
if name == 'tab':
self.seq.append(self.tab)
elif name == 'seq':
consume(self.seq)
def consume(seq):
print 'seq', len(seq) # Do something interesting here
if __name__ == "__main__":
main()
--------------------- consume.py -------------------------
I see value in having a way to consume ausearch output without having
access to audit development libraries. If I want to write a one off
audit analysis tool, the combination of the XML formatter and a simple
Python script would greatly shorten the time required to write the
analysis tool. Having that tool is allowing me to analyze audit data
right now.
The program that converts ausearch output into XML is called auxml,
and is in the CVS repository of the polgen project on SourceForge, in
the pkg/auxml directory of the polgen module. The package includes a
manual page.
John
2:46 p.m.
ramsdell(a)mitre.org (John D. Ramsdell) writes:
Steve Grubb <sgrubb(a)redhat.com> writes:
> Each record is denoted by a type which indicates what fields will
> follow. Information in the fields are held by a name/value pair that
> contains an '=' between them. Each field is separated from one
> another by a space or comma.
Please do not separate fields with commas.
Opps. I think I misunderstood the text. I thought it was defining
ausearch output, but it must be describing something about the data
source. I notice ausearch output always separates fields with a space
and not a comma.
John
11 a.m.
Functions
---------
auparse_state_t - is an opaque data type used for maintaining library
state.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_BUFFER } ausource_t;
auparse_state_t *auparse_init(ausource_t source, const void *b) - allow
init
of library. Set data source: logs, file, buffer. The pointer 'b' is used
to
set the file name or pass the buff when those types are given.
If I have a bunch of collected log files from around the network in my
sysadmins home directory, I want to view all these files together and maybe
with different filters (this is the seaudit GUI). Can we make
auparse_init() support multiple files specified manually?
Example:
char ** files = { "/home/admin/log1", "/home/admin/log2", NULL };
ausearch_init(AUSOURCE_FILES, files)
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD,
AUSEARCH_STOP_FIELD } austop_t;
int ausearch_set_param(auparse_state_t *au, const char *field, const char
*op,
const char *value, austop_t where) - set search
options. The field would be the left hand side of the audit name/value
pairs.
I am a bit confused about the capabilities provided above. Can I make an
array of these auparse_state_t objects and maintain several different search
"views" on the library iterating over each view independently? This would
seem ideal.
The op would be how to match: =,!=,>,<. The value would be the
right hand
side of the audit field name/value pairs. The where parameter tells the
search library where to place the internal cursor when a match is found.
It
could be on first field of first record, first field of record containing
the
match, or the field that matches.
Kevin Carr
Tresys Technology
410.290.1411 x137
11:09 a.m.
On Monday 13 March 2006 12:00, Kevin Carr wrote:
If I have a bunch of collected log files from around the network in
my
sysadmins home directory, I want to view all these files together and maybe
with different filters (this is the seaudit GUI). Can we make
auparse_init() support multiple files specified manually?
I suppose. I'll add that to the spec.
> int ausearch_set_param(auparse_state_t *au, const char *field,
const char
> *op,
> const char *value, austop_t where) - set search
> options. The field would be the left hand side of the audit name/value
> pairs.
I am a bit confused about the capabilities provided above. Can I make an
array of these auparse_state_t objects and maintain several different
search "views" on the library iterating over each view independently? This
would seem ideal.
I think the answer is Yes. Each state would be a search or iteration instance.
They could be searching different files or have different search parameters.
I think the analogy that was used previously was to think of them as "FILE
*". Using that analogy, a program can have multiple FILE *, each unique since
they have their own fopen call which initializes the resources and state.
auparse_init would be equivalent to fopen in this analogy.
-Steve
11:34 a.m.
I think the answer is Yes. Each state would be a search or iteration
instance.
They could be searching different files or have different search
parameters.
I think the analogy that was used previously was to think of them as "FILE
*". Using that analogy, a program can have multiple FILE *, each unique
since
they have their own fopen call which initializes the resources and state.
auparse_init would be equivalent to fopen in this analogy.
It seems that the naming is a bit confusing then. Should it be
ausearch_state_t instead of auparse_state_t, as it is setting information
related to the search. It also makes sense because ausearch_set_param()
should be setting information on a ausearch_state_t. This seems more inline
with the Ground Rules we specified.
Kevin Carr
Tresys Technology
410.290.1411 x137
12:27 p.m.
On Monday 13 March 2006 12:34, Kevin Carr wrote:
It seems that the naming is a bit confusing then. Should it be
ausearch_state_t instead of auparse_state_t, as it is setting information
related to the search.
No, its general information. For example, the list of files has nothing to do
with whether or not a search has been set or simple brute force iteration is
being done.
-Steve
11:12 a.m.
On Monday 13 March 2006 12:00, Kevin Carr wrote:
char ** files = { "/home/admin/log1",
"/home/admin/log2", NULL };
ausearch_init(AUSOURCE_FILES, files)
I wonder if we should have AUSOURCE_FILE_ARRAY and AUSOURCE_BUFFER_ARRAY to
indicate the pointer being passed is a NULL terminated array of filenames or
buffer addresses.
-Steve
1:33 p.m.
On Friday 10 March 2006 16:33, Steve Grubb wrote:
OK. I think the last round of comments was helpful.
Audit Event Parsing Library Specifications
==========================================
Definitions
-----------
An audit event is all records that have the same host, timestamp, and
serial number. Each event on a host has a unique timestamp and serial
number. An event is composed of multiple records which have information about
different aspects of an audit event. Each record is denoted by a type which
indicates what fields will follow. Information in the fields are held by a
name/value pair that contains an '=' between them. Each field is separated
from one another by a space or comma.
Ground Rules
------------
All functions that begin with ausearch are related to searching for a subset
of events based on certain criteria. All functions that begin with auparse
are used to access events, records, and fields sequentially and without
regard to any search options that may be in effect. All functions return 1
on success and 0 on failure unless otherwise noted. Where the return type is
a char pointer, NULL will indicate failure. The data structures will be
hidden from the external application. Access to fields is a name/value style.
You access the fields through functions that either return a pointer to an
immutable, zero-terminated array of ASCII characters or integral values. Every
function (except auparse_init) takes a parameter, au, which is the internal
state information for the current query.
Name-Value Matching Operators
-----------------------------
The ausearch_ functions will select records in its search results based on
operators used for matching name/value pairs. For fields that are numeric,
the following mathematical operators are allowed: =,!=,>,=>,<,<=. The field
is converted to a number before matching is done.
For fields that are non-numeric, the operators in use will be:
= The string completely matches
!= The string does not match
~ A substring match is done
regex Regular expression is used
Functions
---------
auparse_state_t - is an opaque data type used for maintaining library state.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_FILE_ARRAY,
AUSOURCE_BUFFER, AUSOURCE_BUFFER_ARRAY } ausource_t;
auparse_state_t *auparse_init(ausource_t source, const void *b) - allow init
of library. Set data source: logs, file, null terminated array of filenames,
buffer, null terminated array of addresses. The pointer 'b' is used to set
the file name or pass the buff when those types are given.
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD,
AUSEARCH_STOP_FIELD } austop_t;
int ausearch_set_param(auparse_state_t *au, const char *field, const char *op,
const char *value, austop_t where) - set search
options. The field would be the left hand side of the audit name/value pairs.
The op would be the operator described in the section above telling how to
match. The value would be the right hand side of the audit field name/value
pairs. The where parameter tells the search library where to place the
internal cursor when a match is found. It could be on first field of first
record, first field of record containing the match, or the field that
matches.
int ausearch_next_event(auparse_state_t *au) - traverse to the next event that
yields a match based on the given search criteria.
int auparse_next_event(auparse_state_t *au) - traverse to next event. This
allows access to time and serial number.
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
const char *host; // Machine's name
} event_t;
event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time stamp of
current record
time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds of
current record
time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds time of
current record
unsigned long auparse_get_serial(auparse_state_t *au) - retrieve serial number
of current record
const char *auparse_get_node(auparse_state_t *au) - retrieve host (node) name
of current record
int auparse_first_record(auparse_state_t *au) - set iterator to first record
in current event
int auparse_next_record(auparse_state_t *au) - traverse to next record in
event. This allows access to the event type
int auparse_get_type(auparse_state_t *au) - retrieve type of current record
int auparse_first_field(auparse_state_t *au) - set field pointer to first in
current record
int auparse_next_field(auparse_state_t *au) - traverse the fields in a record
const char *auparse_find_field(auparse_state_t *au, const char *name) - find a
given field in a event or record. Name is the left hand side of the name/value
pair. Returns pointer to the value as ascii text.
const char *auparse_find_field_next(auparse_state_t *au) - find the next
occurance of that field in the same record. Returns pointer to the value as
ascii text.
const char *auparse_get_field_str(auparse_state_t *au) - return current field
value as a string
int auparse_get_field_int(auparse_state_t *au) - return current field value
as an int
const char *auparse_interpret_field(auparse_state_t *au) - interpret the
current field
int auparse_destroy(auparse_state_t *au) - free all data structures and close
file descriptors
Code Example
------------
int main(void)
{
auparse_state_t *au = auparse_init(AUSOURCE_LOGS, NULL);
if (au == NULL)
exit(1);
if (!ausearch_set_param(au, "auid", "=", "500",
AUSEARCH_STOP_EVENT))
exit(1);
while (ausearch_next_event(au)) {
if (auparse_find_field(au, "auid")) {
printf("auid=%s\n", auparse_interpret_field(au));
}
}
auparse_destroy(au);
return 0;
}
1:57 p.m.
Hi,
I joined the discussion a little late, so please bear with me for asking
obvious things.
On Monday 13 March 2006 13:33, Steve Grubb wrote:
An audit event is all records that have the same host, timestamp,
and
serial number.
What happens if two events happen on the same time stamp?
What is the time granularity?
Even a millisecond can be a long time for a computer.
Why do we need a serial number?
Information in the fields
are held by a name/value pair that contains an '=' between them. Each
field is separated from one another by a space or comma.
What happens if the data contains a space, comma, or equals sign?
Is quoting allowed? How is it done?
All functions return 1 on success and 0 on failure unless
otherwise noted.
How can an application query reasons for failure?
Is errno set?
You access the
fields through functions that either return a pointer to an immutable,
zero-terminated array of ASCII characters or integral values.
How can you keep the data immutable?
Everybody can cast away the const.
Is this a concern here? Can this introduce problems?
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
const char *host; // Machine's name
} event_t;
event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time
stamp of current record
time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds
of current record
time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds
time of current record
What is the difference between get_timestamp and get_time and get_milli?
int auparse_first_record(auparse_state_t *au) - set iterator to
first
record in current event
int auparse_next_record(auparse_state_t *au) - traverse to next record
in event. This allows access to the event type
Is there something like a has_more_records or will next_record just fail
if there is none?
(In that case it would be especially important to be able to distinguish
between failure and "end of records".)
(Same for iterating the fields in a record.)
const char *auparse_interpret_field(auparse_state_t *au) - interpret
the current field
What does interpreting mean here?
if (!ausearch_set_param(au, "auid", "=",
"500",
AUSEARCH_STOP_EVENT)) exit(1);
Is there a special reason to pass in the comparison operator as a char*
rather than a typedef'd int?
Robert
3:23 p.m.
On Monday 13 March 2006 14:57, Robert Wenner wrote:
On Monday 13 March 2006 13:33, Steve Grubb wrote:
> An audit event is all records that have the same host, timestamp, and
> serial number.
What happens if two events happen on the same time stamp?
Nothing bad happens. They are still unique because of serial numbers which are
atomically incremented in the kernel.
What is the time granularity?
Millisecond
Why do we need a serial number?
To separate events with the same time stamp.
> Information in the fields
> are held by a name/value pair that contains an '=' between them. Each
> field is separated from one another by a space or comma.
What happens if the data contains a space, comma, or equals sign?
If it contains a character that has a delimiter, it is encoded with ascii hex.
Is quoting allowed? How is it done?
I assume you mean escaping. When a field that is under user control is
recorded, it is checked by the audit_log_untrustedstring function. That
function escapes it if needed.
> All functions return 1 on success and 0 on failure unless
> otherwise noted.
How can an application query reasons for failure?
errno
Is errno set?
Yes.
> You access the
> fields through functions that either return a pointer to an immutable,
> zero-terminated array of ASCII characters or integral values.
How can you keep the data immutable?
Everybody can cast away the const.
I suppose you are right. But it won't be an application that is part of the
cert.
Is this a concern here? Can this introduce problems?
To me, no. Just because they mess up their copy of the data doesn't mean they
messed up the data source.
> typedef struct
> {
> time_t sec; // Event seconds
> unsigned int milli; // millisecond of the timestamp
> unsigned long serial; // Serial number of the event
> const char *host; // Machine's name
> } event_t;
>
> event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time
> stamp of current record
> time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds
> of current record
> time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds
> time of current record
What is the difference between get_timestamp and get_time and get_milli?
What they return.
> int auparse_first_record(auparse_state_t *au) - set iterator to
first
> record in current event
>
> int auparse_next_record(auparse_state_t *au) - traverse to next record
> in event. This allows access to the event type
Is there something like a has_more_records or will next_record just fail
if there is none?
Fail if there is none.
> const char *auparse_interpret_field(auparse_state_t *au) -
interpret
> the current field
What does interpreting mean here?
uid=0 becomes uid=root
> if (!ausearch_set_param(au, "auid",
"=", "500",
> AUSEARCH_STOP_EVENT)) exit(1);
Is there a special reason to pass in the comparison operator as a char*
rather than a typedef'd int?
Ease of use from other languages.
-Steve
4:51 p.m.
Another item that came up here at Tresys is the ability to do log
monitoring. After our initial parse/search routine, we would like to be
able to check every so often to see if new messages have been generated and
then display the messages if they match our search criteria. How do you
imagine this fitting in to the API? Can we work this in somehow?
Kevin Carr
Tresys Technology
410.340.3092
On Friday 10 March 2006 16:33, Steve Grubb wrote:
> OK. I think the last round of comments was helpful.
Audit Event Parsing Library Specifications
==========================================
Definitions
-----------
An audit event is all records that have the same host, timestamp, and
serial number. Each event on a host has a unique timestamp and serial
number. An event is composed of multiple records which have information
about
different aspects of an audit event. Each record is denoted by a type
which
indicates what fields will follow. Information in the fields are held by a
name/value pair that contains an '=' between them. Each field is separated
from one another by a space or comma.
Ground Rules
------------
All functions that begin with ausearch are related to searching for a
subset
of events based on certain criteria. All functions that begin with auparse
are used to access events, records, and fields sequentially and without
regard to any search options that may be in effect. All functions return 1
on success and 0 on failure unless otherwise noted. Where the return type
is
a char pointer, NULL will indicate failure. The data structures will be
hidden from the external application. Access to fields is a name/value
style.
You access the fields through functions that either return a pointer to an
immutable, zero-terminated array of ASCII characters or integral values.
Every
function (except auparse_init) takes a parameter, au, which is the
internal
state information for the current query.
Name-Value Matching Operators
-----------------------------
The ausearch_ functions will select records in its search results based on
operators used for matching name/value pairs. For fields that are numeric,
the following mathematical operators are allowed: =,!=,>,=>,<,<=. The
field
is converted to a number before matching is done.
For fields that are non-numeric, the operators in use will be:
= The string completely matches
!= The string does not match
~ A substring match is done
regex Regular expression is used
Functions
---------
auparse_state_t - is an opaque data type used for maintaining library
state.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_FILE_ARRAY,
AUSOURCE_BUFFER, AUSOURCE_BUFFER_ARRAY } ausource_t;
auparse_state_t *auparse_init(ausource_t source, const void *b) - allow
init
of library. Set data source: logs, file, null terminated array of
filenames,
buffer, null terminated array of addresses. The pointer 'b' is used to set
the file name or pass the buff when those types are given.
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD,
AUSEARCH_STOP_FIELD } austop_t;
int ausearch_set_param(auparse_state_t *au, const char *field, const char
*op,
const char *value, austop_t where) - set search
options. The field would be the left hand side of the audit name/value
pairs.
The op would be the operator described in the section above telling how to
match. The value would be the right hand side of the audit field
name/value
pairs. The where parameter tells the search library where to place the
internal cursor when a match is found. It could be on first field of first
record, first field of record containing the match, or the field that
matches.
int ausearch_next_event(auparse_state_t *au) - traverse to the next event
that
yields a match based on the given search criteria.
int auparse_next_event(auparse_state_t *au) - traverse to next event. This
allows access to time and serial number.
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
const char *host; // Machine's name
} event_t;
event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time stamp
of
current record
time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds of
current record
time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds time
of
current record
unsigned long auparse_get_serial(auparse_state_t *au) - retrieve serial
number
of current record
const char *auparse_get_node(auparse_state_t *au) - retrieve host (node)
name
of current record
int auparse_first_record(auparse_state_t *au) - set iterator to first
record
in current event
int auparse_next_record(auparse_state_t *au) - traverse to next record in
event. This allows access to the event type
int auparse_get_type(auparse_state_t *au) - retrieve type of current
record
int auparse_first_field(auparse_state_t *au) - set field pointer to first
in
current record
int auparse_next_field(auparse_state_t *au) - traverse the fields in a
record
const char *auparse_find_field(auparse_state_t *au, const char *name) -
find a
given field in a event or record. Name is the left hand side of the
name/value
pair. Returns pointer to the value as ascii text.
const char *auparse_find_field_next(auparse_state_t *au) - find the next
occurance of that field in the same record. Returns pointer to the value
as
ascii text.
const char *auparse_get_field_str(auparse_state_t *au) - return current
field
value as a string
int auparse_get_field_int(auparse_state_t *au) - return current field
value
as an int
const char *auparse_interpret_field(auparse_state_t *au) - interpret the
current field
int auparse_destroy(auparse_state_t *au) - free all data structures and
close
file descriptors
Code Example
------------
int main(void)
{
auparse_state_t *au = auparse_init(AUSOURCE_LOGS, NULL);
if (au == NULL)
exit(1);
if (!ausearch_set_param(au, "auid", "=", "500",
AUSEARCH_STOP_EVENT))
exit(1);
while (ausearch_next_event(au)) {
if (auparse_find_field(au, "auid")) {
printf("auid=%s\n", auparse_interpret_field(au));
}
}
auparse_destroy(au);
return 0;
}
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
5:01 p.m.
On Monday 13 March 2006 17:51, Kevin Carr wrote:
How do you imagine this fitting in to the API? Can we work this in
somehow?
Call auparse_get_timestamp() on the last record to get all the event
information. Use that as the start position in the next search. That would be
set like
ausearch_set_param(au, "start_time", "=", "3/13/2006
17:57:00",
AUSEARCH_STOP_EVENT);
Then call auparse_get_timestamp() and compare them. If equal goto the next
event like with auparse_next_event(). I am adding the event timestamp
comparison function to the spec since I already have it in ausearch.
-Steve
5:05 p.m.
On Monday 13 March 2006 17:51, Kevin Carr wrote:
Another item that came up here at Tresys is the ability to do log
monitoring.
As an aside...this is not the recommended thing to do since every access of
the audit logs are an auditable event. If you have to do real-time
monitoring, I would suggest using the audit event dispatcher interface. That
gets all audit events in realtime. The parsing specs we are defining right
now also take a buffer as an input source so that they can be used to examine
events passed via the event dispatcher.
After our initial parse/search routine, we would like to be able to
check
every so often to see if new messages have been generated and then display
the messages if they match our search criteria.
This sounds like a 100% fit for the audit event interface.
-Steve
8:50 a.m.
I'd like to suggest a convention for representing name/value pairs
made available through the parsing library. As stated before, I'd
like to be sure that the strings returned as the name and as the value
of a field contain 7-bit ASCII characters excluding zero. The
question is what happens if a value represents binary data. Such a
value might appear in the log if the initial sequence of the second
argument of a write system call is recorded.
I suggest that in the string returned as the name or value of a field,
we let the alphanumeric characters, space, and the following graphic
characters stand for themselves: !#%^*(_)~+=~[]'|;:{},.<>/?$@`. The
characters that can be represented as a single character escape in a C
string would be represented the same way in a value, and the remaining
characters would be represented as a backslash, followed by the letter
'x', followed by two hexidecimal digits. With this specification, a
value could be wrapped in double quotes, and become a C string
constant. Note with this specification, characters such as '\t' and
'\n' would appear quoted, and programs can make use of this fact.
John
Steve Grubb <sgrubb(a)redhat.com> writes:
On Friday 10 March 2006 16:33, Steve Grubb wrote:
> OK. I think the last round of comments was helpful.
Audit Event Parsing Library Specifications
==========================================
...
8:59 a.m.
On Wednesday 15 March 2006 09:50, John D. Ramsdell wrote:
As stated before, I'd like to be sure that the strings returned
as the name
and as the value of a field contain 7-bit ASCII characters excluding
zero.
The name will be 7-bit ASCII, however, I cannot guarantee what the value will
be. It comes from the kernel. If someone in another country uses their native
language to name files and it shows up in the audit record, what should we
do? This subject has come up before and I believe we have to make an effort
to support them. I don't believe CC requires this, our user's do.
The question is what happens if a value represents binary data. Such
a
value might appear in the log if the initial sequence of the second
argument of a write system call is recorded.
I believe 0 still terminates the string.
I suggest that in the string returned as the name or value of a
field,
we let the alphanumeric characters, space, and the following graphic
characters stand for themselves: !#%^*(_)~+=~[]'|;:{},.<>/?$@`. The
characters that can be represented as a single character escape in a C
string would be represented the same way in a value, and the remaining
characters would be represented as a backslash, followed by the letter
'x', followed by two hexidecimal digits. With this specification, a
value could be wrapped in double quotes, and become a C string
constant. Note with this specification, characters such as '\t' and
'\n' would appear quoted, and programs can make use of this fact.
I think the raw values should be available. The escaping could be a library
function just as interpret is a library function.
-Steve
9:12 a.m.
ramsdell(a)mitre.org (John D. Ramsdell) writes:
I'd like to suggest a convention for representing name/value
pairs
made available through the parsing library.
Alternatively, the quoting convention described previously could be
used only for ausearch text output, and users of the parsing library
should be prepared to handle arbitrary character data. That way the
application would choose the quoting convention when and if it
serialized values. In this case, we should remove the restriction
that the values returned by the library contain only US-ASCII
characters.
John
8:26 a.m.
Steve,
I decided to write a simple program against the interface defined by
the Audit Event Parsing Library Specifications, but found a function
missing from the specification. I could find no way to get the name
associated with the current field. Let me suggest the addition of a
function with the following prototype.
const char *auparse_get_field_name(auparse_state_t *au) - return
current field name as a string
With this addition, I'm guessing I can translate a log into a textual
form similar to what is produced by ausearch, but with quoting, by
using a program with a main routine something like the following.
int
main(int argc, char **argv)
{
auparse_state_t *au = auparse_init(AUSOURCE_LOGS, NULL);
while (auparse_next_event(au)) {
if (auparse_first_record(au)) {
printf("---\n");
do { /* for each record in this event */
if (auparse_first_field(au)) {
int not_first = 0;
do { /* for each field in this record */
if (not_first)
putchar(' ');
else
not_first = 1;
const char *name = auparse_get_field_name(au);
const char *value = auparse_interpret_field(au);
if (!name || !value) {
fprintf(stderr, "Internal error\n");
return 1;
}
putitem(name);
putchar('=');
putitem(value); /* Putitem quotes an item as needed. */
} while (auparse_next_field(au));
}
putchar('\n');
} while (auparse_next_record(au));
}
}
return 0;
}
John
Steve Grubb <sgrubb(a)redhat.com> writes:
On Friday 10 March 2006 16:33, Steve Grubb wrote:
> OK. I think the last round of comments was helpful.
Audit Event Parsing Library Specifications
==========================================
Definitions
-----------
An audit event is all records that have the same host, timestamp, and
serial number. Each event on a host has a unique timestamp and serial
number. An event is composed of multiple records which have information about
different aspects of an audit event. Each record is denoted by a type which
indicates what fields will follow. Information in the fields are held by a
name/value pair that contains an '=' between them. Each field is separated
from one another by a space or comma.
Ground Rules
------------
All functions that begin with ausearch are related to searching for a subset
of events based on certain criteria. All functions that begin with auparse
are used to access events, records, and fields sequentially and without
regard to any search options that may be in effect. All functions return 1
on success and 0 on failure unless otherwise noted. Where the return type is
a char pointer, NULL will indicate failure. The data structures will be
hidden from the external application. Access to fields is a name/value style.
You access the fields through functions that either return a pointer to an
immutable, zero-terminated array of ASCII characters or integral values. Every
function (except auparse_init) takes a parameter, au, which is the internal
state information for the current query.
Name-Value Matching Operators
-----------------------------
The ausearch_ functions will select records in its search results based on
operators used for matching name/value pairs. For fields that are numeric,
the following mathematical operators are allowed: =,!=,>,=>,<,<=. The field
is converted to a number before matching is done.
For fields that are non-numeric, the operators in use will be:
= The string completely matches
!= The string does not match
~ A substring match is done
regex Regular expression is used
Functions
---------
auparse_state_t - is an opaque data type used for maintaining library state.
typedef enum { AUSOURCE_LOGS, AUSOURCE_FILE, AUSOURCE_FILE_ARRAY,
AUSOURCE_BUFFER, AUSOURCE_BUFFER_ARRAY } ausource_t;
auparse_state_t *auparse_init(ausource_t source, const void *b) - allow init
of library. Set data source: logs, file, null terminated array of filenames,
buffer, null terminated array of addresses. The pointer 'b' is used to set
the file name or pass the buff when those types are given.
typedef enum { AUSEARCH_STOP_EVENT, AUSEARCH_STOP_RECORD,
AUSEARCH_STOP_FIELD } austop_t;
int ausearch_set_param(auparse_state_t *au, const char *field, const char *op,
const char *value, austop_t where) - set search
options. The field would be the left hand side of the audit name/value pairs.
The op would be the operator described in the section above telling how to
match. The value would be the right hand side of the audit field name/value
pairs. The where parameter tells the search library where to place the
internal cursor when a match is found. It could be on first field of first
record, first field of record containing the match, or the field that
matches.
int ausearch_next_event(auparse_state_t *au) - traverse to the next event that
yields a match based on the given search criteria.
int auparse_next_event(auparse_state_t *au) - traverse to next event. This
allows access to time and serial number.
typedef struct
{
time_t sec; // Event seconds
unsigned int milli; // millisecond of the timestamp
unsigned long serial; // Serial number of the event
const char *host; // Machine's name
} event_t;
event_t auparse_get_timestamp(auparse_state_t *au) - retrieve time stamp of
current record
time_t auparse_get_time(auparse_state_t *au) - retrieve time in seconds of
current record
time_t auparse_get_milli(auparse_state_t *au) - retrieve milliseconds time of
current record
unsigned long auparse_get_serial(auparse_state_t *au) - retrieve serial number
of current record
const char *auparse_get_node(auparse_state_t *au) - retrieve host (node) name
of current record
int auparse_first_record(auparse_state_t *au) - set iterator to first record
in current event
int auparse_next_record(auparse_state_t *au) - traverse to next record in
event. This allows access to the event type
int auparse_get_type(auparse_state_t *au) - retrieve type of current record
int auparse_first_field(auparse_state_t *au) - set field pointer to first in
current record
int auparse_next_field(auparse_state_t *au) - traverse the fields in a record
const char *auparse_find_field(auparse_state_t *au, const char *name) - find a
given field in a event or record. Name is the left hand side of the name/value
pair. Returns pointer to the value as ascii text.
const char *auparse_find_field_next(auparse_state_t *au) - find the next
occurance of that field in the same record. Returns pointer to the value as
ascii text.
const char *auparse_get_field_str(auparse_state_t *au) - return current field
value as a string
int auparse_get_field_int(auparse_state_t *au) - return current field value
as an int
const char *auparse_interpret_field(auparse_state_t *au) - interpret the
current field
int auparse_destroy(auparse_state_t *au) - free all data structures and close
file descriptors
Code Example
------------
int main(void)
{
auparse_state_t *au = auparse_init(AUSOURCE_LOGS, NULL);
if (au == NULL)
exit(1);
if (!ausearch_set_param(au, "auid", "=", "500",
AUSEARCH_STOP_EVENT))
exit(1);
while (ausearch_next_event(au)) {
if (auparse_find_field(au, "auid")) {
printf("auid=%s\n", auparse_interpret_field(au));
}
}
auparse_destroy(au);
return 0;
}
--
Linux-audit mailing list
Linux-audit(a)redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
8:47 a.m.
On Thursday 16 March 2006 09:26, John D. Ramsdell wrote:
I decided to write a simple program against the interface defined by
the Audit Event Parsing Library Specifications, but found a function
missing from the specification. I could find no way to get the name
associated with the current field. Let me suggest the addition of a
function with the following prototype.
Hmm, that was there. I'll update the specs. I put the specs off my people page
a couple days ago. The latest copy is here:
http://people.redhat.com/sgrubb/audit/audit-parse.txt
Thanks for pointing that out. I think there may be 1 or 2 more additions to
the specs as well.
-Steve
8:08 a.m.
New subject: type=SOCKADDR record missing for socketcall(accept)?
Steve,
On a machine running Rawhide, I'm studying the output produced by
ausearch for the socketcall system call. I noticed that a
socketcall(bind) and socketcall(connect) event contain a record of
type=SOCKADDR, but I cannot see one for a system call event associated
with socketcall(accept). Recording the sockaddr of an accepted socket
is important for cross platform information flow analysis.
John
$ uname -a
Linux drawlight.mitre.org 2.6.15-1.2032.2.3_FC5.lspp.12smp #1 SMP Fri Mar 10 15\:44:23 EST
2006 i686 i686 i386 GNU/Linux
9:36 a.m.
New subject: type=SOCKADDR record missing for socketcall(accept)?
On Thursday 23 March 2006 09:08, John D. Ramsdell wrote:
I noticed that a socketcall(bind) and socketcall(connect) event
contain a
record of type=SOCKADDR, but I cannot see one for a system call event
associated with socketcall(accept). Recording the sockaddr of an accepted
socket is important for cross platform information flow analys
Thanks for pointing this out. The following patch should address this.
Signed-off-by: Steve Grubb <sgrubb(a)redhat.com>
diff -urp linux-2.6.15.x86_64.orig/net/socket.c linux-2.6.15.x86_64/net/socket.c
--- linux-2.6.15.x86_64.orig/net/socket.c 2006-03-23 10:16:44.000000000 -0500
+++ linux-2.6.15.x86_64/net/socket.c 2006-03-23 10:27:20.000000000 -0500
@@ -263,6 +263,8 @@ int move_addr_to_user(void *kaddr, int k
return -EINVAL;
if(len)
{
+ if (audit_sockaddr(klen, kaddr))
+ return -ENOMEM;
if(copy_to_user(uaddr,kaddr,len))
return -EFAULT;
}
6847
days inactive
6865
days old
linux-audit@lists.linux-audit.osci.io
79 comments
9 participants
participants (9)
-
Debora Velarde -
John D. Ramsdell -
Kevin Carr -
Klaus Weidner -
LC Bruzenak -
Loulwa Salem -
Michael C Thompson -
Robert Wenner -
Steve Grubb