RE: Linux audit performance impact
I agree that changing the formatting of the records could break the existing applications
that consume them, and I didn't mean changing or eliminating of the formatting
completely.
We agree that formatting is required for logging the records(as buffers) into the log
files.
We are wondering if these records can be made available as RAW records so that the
analytical programs which are capable of reading them for processing can perform better.
This option of RAW mode for the events can be an additional option where, kauditd
delivers
the audit buffer without formatting. Any comments on this?
On Monday, February 16, 2015 11:25:57 AM Viswanath, Logeswari P
wrote:
> I configured the system to audit open system call alone instead of all
>the system calls (our loader program executes) and hence I saw the
> massive improvement in performance. My fix is not causing any change
>in the performance. I wrongly communicated that the fix is causing
>performance improvement. Sorry for that.
>
> As per the perf data, the format_decode is the function where most of
> the time is spent i.e. formatting the record in the buffer before
>delivering the data to user space. We need to eliminate formatting
>records to increase the performance. Any idea why we need to format
>the record and whether can we add an option (RAW) to deliver the
>record without formatting to user space?
Introducing any changes to the format of the record can cause all
analytical programs, both open source and proprietary, to stop working correctly. This
cannot be changed.
I think there is room for improvement however. There are times when strings are being
glued together and a stpcpy works just fine. There are times when a numeric hex conversion
is being done and %x is very slow. Same with %d.
The other issue is that the audit system's philosophy has not been to optimize the
formatting of the event, because events _should_ be rare. Meaning that if you are getting
hundred of events per second, something is seriously wrong with the rules.
It has been optimized to provide as little impact as possible when _not_ generating
events. Meaning that we want it as fast as possible in letting the system operate
normally.
Again, there is room for improvement in both cases of triggering and not triggering
events. But the format of events can't really change without a lot of coordination. I
have a test suite here:
http://people.redhat.com/sgrubb/audit/ausearch-test-0.5.tar.gz
That can check that events are searchable by the main audit utility. If changes cause that
to fail, then its a sign you'll break the whole world.
-Steve