Objective: learn to use DTrace, in conjunction with the OpenGrok
kernel source browser. We will add the kernel debugger "mdb -k" 
to the mix later in the course.

A good slide deck to review DTrace features (I showed it in class):
http://d3s.mff.cuni.cz/teaching/crash_dump_analysis/slides/10-dtrace.pdf
[note that sun.com links are defunct; search for the page name,
 and you might find it on Oracle's port of the Sun blogs. 
 Copies of some key documents, such as the DTrace User Guide, can be found 
 in the course directory or in local/]

In my experience, DTrace is best learned from examples.
Start with kill.d, sshkeysnoop.d, and shellsnoop.d
as suggested in http://www.brendangregg.com/dtrace.html:
   http://brendangregg.com/DTrace/kill.d
   http://brendangregg.com/DTrace/sshkeysnoop.d
   http://brendangregg.com/DTrace/shellsnoop.d

See also ktrace.d (http://dtracebook.com/index.php/Kernel:ktrace.d)
from the http://dtracebook.com/ collection.

These examples illustrate how a follow the activities of a 
process from the kernel by intercepting its syscalls. Run 
these and make sure you understand all the details.  
(E.g.: what is copyinstr() and why is it necessary?
       What is the difference between 'self' and 'this',
       and why are both needed? Etc.)

Aggregation is a major feature of DTrace, which helps
profile applications' performance and the overall
system activities. 

Study the 'DTrace one-liners'
http://brendangregg.com/DTrace/dtrace_oneliners.txt for examples of
aggregation, and run these on your system. Then study
   http://brendangregg.com/DTrace/bitesize.d

--------------- Exercises: ---------------

1. When a process gets created, print the info about its
   parent and 'grandparent' processes, including their
   names and pids.

(Note that since D does not include loops, printing its
   entire parent chain would be tricky. Loops were 
   not included in D on purpose: DTrace designers did
   not want users to be able to accidentally create endless 
   loops in the kernel.)

2. Trace the code path through which signals are delivered
   to processes (e.g., by 'kill'). Find out which parts
   of the proc_t process descriptor are used in the process,
   especially which 'struct proc *' members enable the
   delivery. 

Hint: combine the system call's function boundary tracing of 
      ktrace.d  with kill.d . Use pointer-chasing off the
      curthread 

3. Notice that kill.d contains a (minor) bug. For example,
suspending and unsuspending emacs results in 

root@openindiana:/home/sergey# ./kill.d 
 FROM      COMMAND   SIG TO     RESULT
  940         bash    25 4294966305 0
  991    emacs-gtk    24 4294966305 0
  940         bash    25 4294966305 0

Clearly, 4294966305 is not a valid PID. What is going 
on and how to fix it?
.
.
.
.
.
.
.
Hint: read 'man -s2 kill' (the specification for the kill syscall)
      carefully.

Hint: recall that DTrace variables' default type may not be
      the most natural type for a particular syscall argument
      (or return value).
      
   




