Looking at Illumos (OpenSolaris) process descriptors with MDB.

MDB is the Solaris' "modular debugger". It has a different command
structure than GDB, and it takes practice and exploration to get
used to. However, MDB offers very rich commands for looking at
the internals of a running kernel ("mdb -k", started as root).

Here is my session. My comments start with SB:

sergey@openindiana:~$ sudo mdb -k
Password: 
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs sata sd ip hook neti sockfs arp usba stmf stmf_sbd fctl lofs random idm cpc fcip crypto sppp nfs ufs logindmux ptm ipc ]
> ::help

Each debugger command in mdb
 is structured as follows:

      [ address [, count]] verb [ arguments ... ]
             ^       ^      ^      ^
 the start --+       |      |      +-- arguments are strings which can be
 address can be an   |      |          quoted using "" or '' or
 expression          |      |          expressions enclosed in $[ ]
                     |      |
 the repeat count  --+      +--------- the verb is a name which begins
 is also an expression                 with either $, :, or ::.  it can also
                                       be a format specifier (/ \ ? or =)

For information on debugger commands (dcmds) and walkers, type:

      ::help cmdname ... for more detailed information on a command
      ::dcmds        ... for a list of dcmds and their descriptions
      ::walkers      ... for a list of walkers and their descriptions
      ::dmods -l     ... for a list of modules and their dcmds and walkers
      ::formats      ... for a list of format characters for / \ ? and =

For information on command-line options, type:
                                      
      $ mdb -?      ... in your shell for a complete list of options

> ::dcmds
$<                       - replace input with macro
$<<                      - source macro
$>                       - log session to a file
$?                       - print status and registers
$C                       - print stack backtrace
$G                       - enable/disable C++ demangling support
$M                       - list macro aliases
<SKIP> 

SB: ::ps is a typical walker command. It walks the process table and
    prints a summary line per process. These summaries helpfully include
    the address of each proc_t structure (process descriptor):

> ::ps
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R      0      0      0      0      0 0x00000001 fffffffffbc2e330 sched
R      3      0      0      0      0 0x00020001 ffffff014e183028 fsflush
R      2      0      0      0      0 0x00020001 ffffff014e184020 pageout
R      1      0      0      0      0 0x4a004000 ffffff014e08f018 init
R   4208      1   4207   4207      0 0x42000000 ffffff0160825060 sshd
R   4195      1   4178   4178      0 0x4a004900 ffffff01609db0b8 modunload
R   4176      1   4159   4159      0 0x4a004900 ffffff01609d20d0 modunload
R   3981      1   3886   3886      0 0x4a004900 ffffff01606a8048 modunload
R   4153      1   4153   4153      0 0x52010000 ffffff015488f010 sendmail
R   4146      1   4146   4146     25 0x52010000 ffffff01609d70c0 sendmail
R   4142      1   4142   4142      0 0x42000000 ffffff0160828058 nscd
R   4128      1   4128   4128      0 0x42000000 ffffff01546b10b0 automountd
R   4129   4128   4128   4128      0 0x42000000 ffffff0153902008 automountd
R   1441      1   1440   1440    101 0x4a004000 ffffff015ff99028 gam_server
R   1422      1   1165   1165    101 0x4a014000 ffffff015b4c2018 firefox
R   1435   1422   1165   1165    101 0x4a014000 ffffff015ae0a008 run-mozilla.sh
R   1439   1435   1165   1165    101 0x4a004000 ffffff015b4bd020 firefox-bin
R   1270      1   1165   1165    101 0x4a004000 ffffff01598270b8 gnome-terminal
R   4156   1270   4156   4156    101 0x4a014000 ffffff0160818078 bash
<SKIP>

SB: Let's look at one proc_t, pick the last line above. 
    Printing memory at ffffff0160818078 (it's 64 bit system)
    interpreting its contents as a struct proc_t:

> ffffff0160818078::print proc_t
{
    p_exec = 0xffffff0158c36c80
    p_as = 0xffffff016054f940
    p_lockp = 0xffffff014ab878c0
    p_crlock = {
        _opaque = [ 0 ]
    }
    p_cred = 0xffffff0156ba4810
    p_swapcnt = 0
    p_stat = '\002'
    p_wcode = '\0'
    p_pidflag = 0
    p_wdata = 0
    p_ppid = 0x4f6
    p_link = 0
    p_parent = 0xffffff01598270b8
    p_child = 0xffffff01609de0b0
    p_sibling = 0xffffff0160817080
    p_psibling = 0
    p_sibling_ns = 0
    p_child_ns = 0
    p_next = 0xffffff015488f010
    p_prev = 0xffffff01609d20d0
<SKIP>
       u_start = {                   
            tv_sec = 2013 Jan 18 19:35:35
            tv_nsec = 0x29b0b934
        }
        u_ticks = 0x1b08bb1
        u_comm = [ "bash" ]
        u_psargs = [ "bash" ]
        u_argc = 0x1
        u_argv = 0x8047abc
        u_envp = 0x8047ac4
        u_cdir = 0xffffff0158eee800
        u_rdir = 0
        u_mem = 0x1463
        u_mem_max = 0x17aa
<SKIP>

SB: The PID of the process sits in a small 'stuct pid', pointed to
    from proc_t. "! grep pid" will help us filter relevant lines:

> ffffff0160818078::print proc_t !grep pid
    p_pidflag = 0
    p_ppid = 0x4f6
    p_pidp = 0xffffff0162365030
    p_lwpid = 0x1
    p_t1_lgrpid = 0
    p_tr_lgrpid = 0xffffffff
    p_ancpid = 0x4f6

SB: Now to print that struct:

> 0xffffff0162365030::print 'struct pid'
{
    pid_prinactive = 0
    pid_pgorphaned = 0x1
    pid_padding = 0
    pid_prslot = 0x63
    pid_id = 0x103c
    pid_pglink = 0xffffff0160818078
    pid_pgtail = 0xffffff0160818078
    pid_link = 0
    pid_ref = 0x4
}

SB: So the PID is 0x103c. Let's cross-check that with regular "ps". 
    Luckily, we need not even leave the MDB shell: to run Unix shell
    commands from it we just need to prefix them with '!'

> !printf "%d\n" 0x103c
4156
> !ps ax | grep bash
  1422 ?        S  0:00 /bin/bash /usr/bin/firefox
  1435 ?        S  0:00 /bin/bash /usr/lib/firefox/run-mozilla.sh /usr/lib/fire
  1273 pts/1    S  0:00 bash
  4212 pts/1    R  0:00 bash -c ps ax | grep bash
  3595 pts/2    S  0:00 bash
  3598 pts/2    S  0:00 sudo bash
  3599 pts/2    S  0:00 bash
  4156 pts/3    S  0:00 bash

SB: Indeed, the last shell it is.

SB: Note that we can follow any pointers and print memory that they
    point to, as long as we know the type of the values or structs
    contained there. We can also print type definitions. 
    Option '-a' prints offsets of each member of a struct type:

> ::print -a kthread_t

0 {
    0 t_link 
    8 t_stk 
    10 t_startpc 
    18 t_bound_cpu 
    20 t_affinitycnt 
    22 t_bind_cpu 
    24 t_flag 
    26 t_proc_flag 
    28 t_schedflag 
    2a t_preempt 
    2b t_preempt_lk 
    2c t_state 
    30 t_pri 
    32 t_epri 
    34 t_cpri 
    36 t_writer 
    37 t_bindflag 
    38 t_pcb {
        38 val 
    }
<SKIP>
    170 t_forw 
    178 t_back 
    180 t_thlink 
    188 t_lwp 
    190 t_procp 

SB: That last one is a pointer to the proc_t that a process belongs to.

SB: =K can be used to get the value of a symbol, i.e., the address
    that this symbol is known to reside at.
    (Why 'K'? See ::formats . Addresses are 8 bytes long. For a 32 bit system
     we'd use 'X', which on this 64bit machine just gives us the lower 4 bytes
     of the address, not very useful)
 
> getpid=K
                fffffffffba44168 
> getpid=X                            
                fba44168        


SB: Disassemble kernel code at that address:   

> getpid::dis 
getpid:                         pushq  %rbp
getpid+1:                       movq   %rsp,%rbp
getpid+4:                       subq   $0x10,%rsp

SB: This is how curthread from /uts/common/sys/thread.h is inlined:
    (the assembly implementation of threadp() is in /uts/intel/asm/thread.h,
     specific to Intel architectures; there are others for Sparc, etc.)

getpid+8:                       movq   %gs:0x18,%rax

SB: And this is the first line of getpid() from /uts/common/syscall/getpid.c,
    p = ttoproc(curthread), 
    and ttoproc(x) is macro-defined as ((x)->t_procp) in thread.h.
    Note that 0x190 is exactly the offset of t_procp in kthread_t.

    Convince yourself that the other offsets in this chain of 
    pointer derefences (C's '->' operator) make sense. Look at the
    code of getpid() if you get confused by assembly logic, but
    don't worry about the "ZONE" branch yet.

getpid+0x11:                    movq   0x190(%rax),%r9
getpid+0x18:                    movq   0xb0(%r9),%r8
getpid+0x1f:                    movl   0x4(%r8),%eax
getpid+0x23:                    movl   %eax,-0x8(%rbp)
getpid+0x26:                    testl  $0x400,0xcc(%r9)
getpid+0x31:                    jne    +0x9     <getpid+0x3c>
getpid+0x33:                    movl   0x34(%r9),%eax
getpid+0x37:                    movl   %eax,-0x4(%rbp)
getpid+0x3a:                    jmp    +0x2c    <getpid+0x68>
getpid+0x3c:                    movq   %gs:0x18,%rax
getpid+0x45:                    movq   0x190(%rax),%r8
getpid+0x4c:                    movq   0x620(%r8),%r8
getpid+0x53:                    movq   0x150(%r8),%r8
getpid+0x5a:                    movq   0xb0(%r8),%r8
getpid+0x61:                    movl   0x4(%r8),%eax
getpid+0x65:                    movl   %eax,-0x4(%rbp)
getpid+0x68:                    movq   -0x8(%rbp),%rax
getpid+0x6c:                    leave  
getpid+0x6d:                    ret    

SB: Raw address would work, too. Note that the disassembler looks up
    the symbol at that address, hits "getpid", and uses that for the
    listing. Symbol lookup can be done with the "::nm" command, too.

> fffffffffba44168::dis
getpid:                         pushq  %rbp
getpid+1:                       movq   %rsp,%rbp
getpid+4:                       subq   $0x10,%rsp
getpid+8:                       movq   %gs:0x18,%rax
getpid+0x11:                    movq   0x190(%rax),%r9
getpid+0x18:                    movq   0xb0(%r9),%r8
<SKIP>

> fffffffffba44168::nm 
Value              Size               Type  Bind  Other Shndx    Name
0xfffffffffba44168|0x000000000000006e|FUNC |GLOB |0x0  |3       |getpid

SB: So you can navigate the process table, in particular visit the proc_t
    descriptors of parent processes, child processes, etc. Some other
    special commands that interpret memory are on page 156 of the textbook
    (Section 2.13).

> ffffff0160818078::pmap
             SEG             BASE     SIZE      RES PATH
ffffff0154899820 0000000008044000      16k      16k [ anon ]
ffffff04b96a0938 0000000008050000     740k     724k /usr/bin/bash
ffffff04b96a08d8 0000000008118000      44k      44k /usr/bin/bash
ffffff04b96a0998 0000000008123000     220k     196k [ anon ]
ffffff0154648d78 00000000fe7d0000      56k      56k /lib/libsocket.so.1
ffffff0162342af8 00000000fe7ee000       4k       4k /lib/libsocket.so.1
ffffff0161167898 00000000feaf0000     456k     416k /lib/libnsl.so.1
ffffff04b96a0c98 00000000feb72000       8k       8k /lib/libnsl.so.1
ffffff015ffca690 00000000feb74000      20k      12k [ anon ]
ffffff015ffa0ea0 00000000fecf0000      64k      16k [ anon ]
ffffff016233e140 00000000fed03000       4k       4k [ anon ]
ffffff0162350f00 00000000fed0c000      76k      32k /usr/lib/locale/en_US.UTF-8/
ffffff01611672f8 00000000fed20000      24k      12k [ anon ]
ffffff015502dd40 00000000fed30000       4k       4k [ anon ]
ffffff04b9738c48 00000000fed40000     180k     140k /lib/libcurses.so.1
ffffff0162342978 00000000fed7d000      28k      28k /lib/libcurses.so.1
ffffff015ff9d8a8 00000000fed84000       8k       8k [ anon ]
ffffff015ff9dde8 00000000fed90000    1216k     940k /usr/lib/libc/libc_hwcap1.so
ffffff015ff85e08 00000000feec0000      36k      36k /usr/lib/libc/libc_hwcap1.so
ffffff04b9738d68 00000000feec9000       8k       8k [ anon ]
ffffff04b973b340 00000000feed0000      24k      24k /lib/libgen.so.1
<SKIP>

> ffffff0160818078::pfiles
FD   TYPE            VNODE INFO
   0  CHR ffffff01608c8040 /dev/pts/3 
   1  CHR ffffff01608c8040 /dev/pts/3 
   2  CHR ffffff01608c8040 /dev/pts/3 
   3 DOOR ffffff04b96a5640 /var/run/name_service_door [door to 'nscd' (proc=
ffffff0160828058)]
 255  CHR ffffff01608c8040 /dev/pts/3 

> ::ptree
fffffffffbc2e330  sched
     ffffff014e183028  fsflush
     ffffff014e184020  pageout
     ffffff014e08f018  init
          ffffff0160825060  sshd
          ffffff01609db0b8  modunload
          ffffff01609d20d0  modunload
          ffffff01606a8048  modunload
          ffffff015488f010  sendmail
          ffffff01609d70c0  sendmail
          ffffff0160828058  nscd
          ffffff01546b10b0  automountd
               ffffff0153902008  automountd
          ffffff015ff99028  gam_server
          ffffff015b4c2018  firefox
               ffffff015ae0a008  run-mozilla.sh
                    ffffff015b4bd020  firefox-bin
          ffffff01598270b8  gnome-terminal
               ffffff0160818078  bash
                    ffffff01609de0b0  emacs-gtk
               ffffff0160817080  bash            <<<<<<<< Look here, our shell
                    ffffff016054d038  sudo
                         ffffff016054b040  bash
<SKIP>

Now you can explore further up and down the tree and around the textbook's 
Chapter 2.

