Hi Steve,

Since auditd.service is disabled nothing is writing to the audit log:

# aureport --start yesterday --event --summary -i

Event Summary Report
======================
total  type
======================
<no events of interest were found>

However, I can parse the audit logs that do make it to the kernel ring buffer (not suppressed by printk):
$ dmesg | grep -Eo 'type=[0-9]+' | sort | uniq -c | sort -n
   1198 type=1400

In the past, before landing on our current (partial) solution to disable the kernel audit subsystem w/ `auditctl -e 0` we tried just filtering out type 1400 messages w/ `auditctl -a exclude,always -F msgtype=1400`. However, that only lead us to seeing other types of audit messages in the kernel ring buffer:
$ dmesg | grep -Eo 'type=[0-9]+' | sort | uniq -c | sort -n
7 type=1109
10 type=1110
12 type=1123
52 type=1112
91 type=1131
93 type=1130
268 type=1105
356 type=1104
357 type=1106
414 type=1006
454 type=1103
459 type=1101

Note, that was with the kernel audit subsystem enabled so perhaps the behavior is different than now where it's disabled entirely.

Regards,
Samuel Bahr
Pinterest Sr. Site Reliability Engineer

On Tue, Jul 25, 2023 at 12:05 PM Steve Grubb <sgrubb@redhat.com> wrote:
On Monday, July 24, 2023 5:06:02 PM EDT Samuel Bahr wrote:
> `auditctl -D` does not make it go away (outputs `No rules`). auditd isn't
> running at all and this behavior is happening purely from the kernel. These
> systems were never set to enabled 2 (locked).
>
> I went ahead and filed a Github issue for this thread:
> https://github.com/linux-audit/audit-kernel/issues/146
>
> The maintainer there suggested it's too difficult to debug due to eBPF
> programs + AWS's modified kernel.

I think there is data that could help decide where the problem might be. On
one of the systems that is still logging, try running an event type report:

aureport --start yesterday --event --summary -i

This should identify what kind of event is being emitted. Based on that, it
might point to where the problem is.

> I've resigned to asking Red Canary to support eBPF mode with `audit=0`
> kernel parameter in their Linux EDR. Let me know if you have any other
> ideas.

I'd say collecting summary information about what kind of events are being
logged would be a good start.

-Steve