4:53 a.m.
Hello All,
The current rate limiter, audit_set_rate_limit limits all types of events.
In our case, we want to limit auditd events with a specific key, as they
are very noisy and consume very high CPU.
From my understanding, this support is currently missing in AuditD.
Is my understanding correct?
--
Anurag Aggarwal
8:11 a.m.
On Tue, Feb 28, 2023 at 5:53 AM Anurag Aggarwal
<anurag19aggarwal(a)gmail.com> wrote:
Hello All,
The current rate limiter, audit_set_rate_limit limits all types of events. In our case,
we want to limit auditd events with a specific key, as they are very noisy and consume
very high CPU.
From my understanding, this support is currently missing in AuditD.
Is my understanding correct?
Hello.
Limiting of audit records is actually done in the kernel, and
currently the rate limit applies equally[1] to all records, there is
no ability to enforce limits per-key. If you have a particular audit
rule which is too verbose *and* you are willing to lose audit records
from that filter rule (which is what would happen if they were rate
limited), you might want to consider making that audit filter rule
more targeted to the event you are interested in logging. Generating
more audit records than you want to see can be a sign of an overly
general audit rule.
Good luck!
[1] Audit records generated by auditd/auditctl are exempt from rate
limiting to help prevent lockups/contention.
--
paul-moore.com
9:35 a.m.
Hello Paul,
Thank you for your information.
If you have a particular audit
rule which is too verbose *and* you are willing to lose audit records
from that filter rule (which is what would happen if they were rate
limited), you might want to consider making that audit filter rule
more targeted to the event you are interested in logging. Generating
more audit records than you want to see can be a sign of an overly
general audit rule.
I agree that having rules which are too verbose is not a very good idea.
Beside this, is there any other mechanism which we can use to get a similar
effect?
--
Anurag Aggarwal
10:31 a.m.
On Tue, Feb 28, 2023 at 10:35 AM Anurag Aggarwal
<anurag19aggarwal(a)gmail.com> wrote:
Hello Paul,
Thank you for your information.
> If you have a particular audit
> rule which is too verbose *and* you are willing to lose audit records
> from that filter rule (which is what would happen if they were rate
> limited), you might want to consider making that audit filter rule
> more targeted to the event you are interested in logging. Generating
> more audit records than you want to see can be a sign of an overly
> general audit rule.
I agree that having rules which are too verbose is not a very good idea.
Beside this, is there any other mechanism which we can use to get a similar effect?
Nothing comes quickly to mind, perhaps others on the mailing list
might have some ideas ... ?
--
paul-moore.com
11:03 a.m.
On 2/28/23 09:31, Paul Moore wrote:
On Tue, Feb 28, 2023 at 10:35 AM Anurag Aggarwal
<anurag19aggarwal(a)gmail.com> wrote:
> Hello Paul,
>
> Thank you for your information.
>
>> If you have a particular audit
>> rule which is too verbose *and* you are willing to lose audit records
>> from that filter rule (which is what would happen if they were rate
>> limited), you might want to consider making that audit filter rule
>> more targeted to the event you are interested in logging. Generating
>> more audit records than you want to see can be a sign of an overly
>> general audit rule.
> I agree that having rules which are too verbose is not a very good idea.
>
> Beside this, is there any other mechanism which we can use to get a similar effect?
Nothing comes quickly to mind, perhaps others on the mailing list
might have some ideas ... ?
Not much else to offer above what Paul already replied. Maybe if we saw
your rule we could offer more.
What we do not know is - do you have any filtering criteria in mind not
covered by the available auditctl exclusions or do you just want to
"sample" randomly?
If the latter, why bother auditing this with a rule at all? You might be
able to remove the rule causing the events and do something in userspace
to audit only what you really want.
Without a bit more context on the events, rule and intent it is hard to
suggest alternatives. But in general, it is preferable to exclude as
much noise as possible in your collection to ensure you get only what is
required/desired in your audit logs.
LCB
--
Lenny Bruzenak
MagitekLTD
9:31 a.m.
What we do not know is - do you have any filtering criteria in mind not
covered by the available auditctl exclusions or do you just want to
"sample" randomly?
If the latter, why bother auditing this with a rule at all? You might be
able to remove the rule causing the events and do something in userspace
to audit only what you really want.
We want to sample system calls like rename.
In many cases, we have seen this overburden and increase auditd cpu
consumption.
In such cases, we want to drop some events randomly, so as to keep cpu
consumption under control.
There are other rules also, for example monitoring login/logout.
For such rules we do not want to drop any event.
--
Anurag Aggarwal
11:05 a.m.
On 3/1/23 08:31, Anurag Aggarwal wrote:
What we do not know is - do you have any filtering criteria in
mind not
covered by the available auditctl exclusions or do you just want to
"sample" randomly?
If the latter, why bother auditing this with a rule at all? You
might be
able to remove the rule causing the events and do something in
userspace
to audit only what you really want.
We want to sample system calls like rename.
In many cases, we have seen this overburden and increase auditd cpu
consumption.
In such cases, we want to drop some events randomly, so as to keep cpu
consumption under control.
There are other rules also, for example monitoring login/logout.
For such rules we do not want to drop any event.
OK; that makes sense.The concern is that when some of the events that
can overflow (rename syscalls in this case) are flooding your auditd,
the ones you care about (login/logout) could be dropped.
But _some_ renames are ok. Maybe you can decide which directories are OK
to have renames under them and exclude those in your key rule? Or only
watch for renames inside certain dirs?
Or if selinux is in force, create policy for the events you definitely
want, then look for those types (either subject or object) in your rule.
This is something I've seen before, where renames that are desired to be
audited use the provided system tools, but for locally developed
application code, they are made to run inside a certain type of a custom
executable and then that type is excluded from the rename syscall rule.
Ideally, the code which is written would self-audit a 1-liner like "I am
going to rename every file under dir /opt/special/stuff/" using
audit_log_user_message so you still have some idea what is happening (if
you care).
Then your "my-rename" program subject type of my_rename_t can be used as
an exclude on the rule. Of course, the caller must then know to use this
rather than the standard utilities.
HTH,
LCB
--
Lenny Bruzenak
MagitekLTD
11:13 p.m.
Or if selinux is in force, create policy for the events you definitely
want, then look for those types (either subject or object) in your rule.
This is something I've seen before, where renames that are desired to be
audited use the provided system tools, but for locally developed
application code, they are made to run inside a certain type of a custom
executable and then that type is excluded from the rename syscall rule.
Ideally, the code which is written would self-audit a 1-liner like "I am
going to rename every file under dir /opt/special/stuff/" using
audit_log_user_message so you still have some idea what is happening (if
you care).
Then your "my-rename" program subject type of my_rename_t can be used as
an exclude on the rule. Of course, the caller must then know to use this
rather than the standard utilities.
This sounds useful and might solve our problem, will it be possible to
share some examples on how this can be achieved?
--
Anurag Aggarwal
11:24 a.m.
On 3/1/23 22:13, Anurag Aggarwal wrote:
Or if selinux is in force, create policy for the events you
definitely want, then look for those types (either subject or
object) in your rule. This is something I've seen before, where
renames that are desired to be audited use the provided system
tools, but for locally developed application code, they are made
to run inside a certain type of a custom executable and then that
type is excluded from the rename syscall rule. Ideally, the code
which is written would self-audit a 1-liner like "I am going to
rename every file under dir /opt/special/stuff/" using
audit_log_user_message so you still have some idea what is
happening (if you care).
Then your "my-rename" program subject type of my_rename_t can be
used as an exclude on the rule. Of course, the caller must then
know to use this rather than the standard utilities.
This sounds useful and might solve our problem, will it be possible to
share some examples on how this can be achieved?
Replying off-list as it is not specifically audit-focused. See Paul, I
CAN learn. 😁
LCB
--
Lenny Bruzenak
MagitekLTD
1:11 p.m.
On Thu, Mar 2, 2023 at 12:24 PM Lenny Bruzenak <lenny(a)magitekltd.com> wrote:
On 3/1/23 22:13, Anurag Aggarwal wrote:
>
> Or if selinux is in force, create policy for the events you definitely want, then
look for those types (either subject or object) in your rule. This is something I've
seen before, where renames that are desired to be audited use the provided system tools,
but for locally developed application code, they are made to run inside a certain type of
a custom executable and then that type is excluded from the rename syscall rule. Ideally,
the code which is written would self-audit a 1-liner like "I am going to rename every
file under dir /opt/special/stuff/" using audit_log_user_message so you still have
some idea what is happening (if you care).
>
> Then your "my-rename" program subject type of my_rename_t can be used as an
exclude on the rule. Of course, the caller must then know to use this rather than the
standard utilities.
This sounds useful and might solve our problem, will it be possible to share some
examples on how this can be achieved?
Replying off-list as it is not specifically audit-focused. See Paul, I CAN learn. 😁
;)
--
paul-moore.com
5:53 a.m.
Limiting of audit records is actually done in the kernel, and
currently the rate limit applies equally[1] to all records, there is
no ability to enforce limits per-key.
One question Paul, will it be ok, if we contribute something similar to the
Auditd Kernel repository?
--
Anurag Aggarwal
9:51 a.m.
On Wednesday, March 8, 2023 6:53:39 AM EST Anurag Aggarwal wrote:
> Limiting of audit records is actually done in the kernel, and
> currently the rate limit applies equally[1] to all records, there is
> no ability to enforce limits per-key.
One question Paul, will it be ok, if we contribute something similar to the
Auditd Kernel repository?
I'm not Paul...but I think what you are proposing is a per rule service
class. Always and best effort where best effort gets discarded when the
backlog is above some heuristic. And rules not saying anything are assumed
always for backwards compatibility. The main issue is that rules are defined
here:
https://github.com/linux-audit/audit-kernel/blob/main/include/uapi/linux/
audit.h#L510
There just really isn't room to add more thinkgs without some userspace API
problem. (This would definitely need a feaure bitmap so user space can make
sense of it.)
I suppose we could declare some bits in flags to carry this meaning? Anyways,
maybe others might chime in to say if they want/need such a feature.
-Steve
11:04 a.m.
On Wed, Mar 8, 2023 at 6:53 AM Anurag Aggarwal
<anurag19aggarwal(a)gmail.com> wrote:
> Limiting of audit records is actually done in the kernel, and
> currently the rate limit applies equally[1] to all records, there is
> no ability to enforce limits per-key.
One question Paul, will it be ok, if we contribute something similar to the Auditd Kernel
repository?
I don't like telling people *not* to work on improvements to the
kernel, I'm happy to see more contributors, especially in the audit
space :)
However, I am fairly skeptical that we could add per-key rate limiting
without introducing a non-trivial amount of overhead to record
generation, which would be a show stopper for this feature given its
expected limited appeal.
--
paul-moore.com
12:16 a.m.
However, I am fairly skeptical that we could add per-key rate limiting
without introducing a non-trivial amount of overhead to record
generation, which would be a show stopper for this feature given its
expected limited appeal.
I understand the reservation. I will spend some time to analyze it impact
Steve, about your comment:
There just really isn't room to add more thinkgs without some
userspace API
problem. (This would definitely need a feaure bitmap so user space can make
sense of it.)
I was not aware that this could cause problems in the userspace API.
Key based filtering for rate limiting could be a useful feature. It is
something that would
help us a lot.
--
Anurag Aggarwal
653
days inactive
662
days old
linux-audit@lists.linux-audit.osci.io
14 comments
4 participants
participants (4)
-
Anurag Aggarwal -
Lenny Bruzenak -
Paul Moore -
Steve Grubb