Kangkook,

Perhaps you can re-test, but modify the kernel boot parameters to
include audit=1 as an additional argument.

Reading the auditd(8) manual, one sees

      A boot param of audit=1 should be added to ensure that all
      processes that run before the audit daemon starts is marked as
      auditable by the kernel. Not doing that will make a few
      processes impossible to properly audit.

On Sep 13, 2015, at 11:58 AM, Richard Guy Briggs <rgb@redhat.com> wrote:

On 15/09/11, Kangkook Jee wrote:

Hi Richard,

Hi Kangkook,

I also did the same experiment for the latest distributions of Fedora
core and Debian and here’s the results.

Fedora-22 (64-bit, 4.0.4-301.fc22.x86_64): Problem reproduced.
Debian-8 (64-bit, 3.16.0-4-amd64): Problem reproduced

Btw, Burn Alting (burn@swtf.dyndns.org) suggested me to append audit=1
to kernel flag. I added the option to boot-loader (grub) and problem
went away. 

On all systems?  This is expected behaviour.  Sorry I was not more
explicit in asking you to test that.  I guess it was implied by asking
what the settings for the kernel command line were.

Now the surprising bit is that CentOS does not demonstrate the problem
without audit=1 in the command line, which leads me to wonder if they
have set "u32             audit_enabled = 1;" around line 83 of
kernel/audit.c in their kernel source.  It would surprise me if they
did, but it would not be completely unreasonable.

Regards, Kangkook

On Sep 11, 2015, at 12:24 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
On 15/09/11, Kangkook Jee wrote:

From the previous reply, I think I misunderstood your question regarding kernel command line. 
Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. 

Ubuntu 14.04 (64-bit): 
  BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7

Ubuntu 12.04 (64-bit): 
  BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash

CentOS 7 (64-bit): 
  BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8

CentOS 6 (64-bit): 
  ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

I don’t see any audit=<value> entries from all examples above. 

Yes, this is what I was seeking from you.  And you are correct, none of
them have audit=1 as I was hoping from at least CentOS.  There is a
chance that the CentOS kernel was compiled with audit=1 hardcoded, but I
think that is a pretty small chance...

I'll have to look at this closer...  But any Debian and Fedora data
points that you can provide would certainly be useful.

/Kangkook

On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb@redhat.com> wrote:

On 15/09/10, Kangkook Jee wrote:

Hi all,

I debugged a bit further to identify distributions that are affected by the issue.
I repeated the same experiment with sshd from 3 more distributions.

CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced

For each of these examples, what is the value of the kernel command line
"audit=<value>" if it is even present?  It is possible that the CentOS
examples include "audit=1" while Ubuntu omits the line.  "cat
/proc/cmdline" should tell you the answer.

After all, Ubuntu family are affected by the issue and I could confirm
that results are inconsistent across two different distribution
families. 

I would be curious what your results are with a recent Debian and with a
recent Fedora.

If you can let us know how can we workaround the issue, it will be a great help.

Regards, Kangkook

On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com> wrote:

Dear all,

We are developing custom user space audit agent to gather system wide system
call trace. While experimenting with various programs, we found out that
processes (daemons) that started early (along with the system bootstrapping) do
not report any audit events at all. These processes typically fall into PID
range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.

1. Reboot the system

2. Add and enable audit events
# /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
       -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
       -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
       -S listen -S socket -S socketpair
# /sbin/auditctl -e1 -b 102400

3. Connect to the system via ssh
 Audit messages generated only from child processes and none are seen from
 the original daemon.

4. Restart sshd 
 # restart ssh

5. Connect again to the system via ssh
Now, we see audit messages from both parent and child processes.

I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
version 3.13.0-58-generic).

I first wonder whether this is intended behavior of audit framework or
not. If it is intended, I also want to know how can we configure auditd
differently to capture system calls from all processes. 

Thanks a lot for your help in advance!

Regards, Kangkook

- RGB

- RGB

- RGB

--
Richard Guy Briggs <rbriggs@redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545