Re: Linux audit performance impact
On Tuesday, February 17, 2015 01:10:21 PM Viswanath, Logeswari P wrote:
I agree that changing the formatting of the records could break the
existing
applications that consume them, and I didn't mean changing or eliminating
of the formatting completely. We agree that formatting is required for
logging the records(as buffers) into the log files. We are wondering if
these records can be made available as RAW records so that the analytical
programs which are capable of reading them for processing can perform
better.
There are no analytical programs that can consume them. :-) I'd like to see
exactly what the bottleneck was and the correction you made. Again, this is an
optimization for something that should rarely happen. Or if it does, its less
than 10 a second. Additionally, the open use case is about the worst
performing one besides connect or accept because of the large amounts of data
that could be generated. Also, kill can generate 1000's of records in one
syscall.
So, I'd like to see what was optimized to see if you tweaked just this one
syscall and how different it might be for analytical programs.
-Steve
This option of RAW mode for the events can be an additional option
where, kauditd delivers the audit buffer without formatting. Any comments
on this?
>On Monday, February 16, 2015 11:25:57 AM Viswanath, Logeswari P wrote:
>
>> I configured the system to audit open system call alone instead of all
>>
> >the system calls (our loader program executes) and hence I saw the
> >
>> massive improvement in performance. My fix is not causing any change
>>
> >in the performance. I wrongly communicated that the fix is causing
> >performance improvement. Sorry for that.
> >
> >
>> As per the perf data, the format_decode is the function where most of
>> the time is spent i.e. formatting the record in the buffer before
>>
> >delivering the data to user space. We need to eliminate formatting
> >records to increase the performance. Any idea why we need to format
> >the record and whether can we add an option (RAW) to deliver the
> >record without formatting to user space?
>Introducing any changes to the format of the record can cause all
>analytical programs, both open source and proprietary, to stop working
>correctly. This cannot be changed.
>I think there is room for improvement however. There are times
when strings
>are being glued together and a stpcpy works just fine. There are times
>when a numeric hex conversion is being done and %x is very slow. Same with
>%d.
>The other issue is that the audit system's philosophy has not
been to
>optimize the formatting of the event, because events _should_ be rare.
>Meaning that if you are getting hundred of events per second, something is
>seriously wrong with the rules.
>It has been optimized to provide as little impact as possible
when _not_
>generating events. Meaning that we want it as fast as possible in letting
>the system operate normally.
>Again, there is room for improvement in both cases of triggering
and not
>triggering events. But the format of events can't really change without a
>lot of coordination. I have a test suite here:
>http://people.redhat.com/sgrubb/audit/ausearch-test-0.5.tar.gz
>
>That can check that events are searchable by the main audit utility. If
>changes cause that to fail, then its a sign you'll break the whole world.
>
>-Steve