After digging deeper, it seems that the spikes correlate perfectly with the "driver" process for e3c. e3c is part of a four-way trunk group running to a Cisco 6509. Any thoughts about what might make one NIC need so much attention?
I have interleaved the ps commands about where they occurred while a sysstat was running. I hope this is clear
F840C*> ps -z CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 1% 88 52 0 140 237 40 24 76 0 0 27 100% 7% : 4% 6% 49 202 0 251 57 408 3173 0 0 0 27 98% 0% - 38% 3% 33 253 0 286 205 1030 684 16 0 0 27 99% 0% - 9% 4% 89 188 0 277 152 506 4188 0 0 0 27 95% 0% - 41% 2% 127 144 0 271 151 948 837 0 0 0 27 99% 0% - 14% 3% 119 2 0 121 102 32 32 16 0 0 27 100% 0% - 1% 0% 24 0 0 24 123 28 0 0 0 0 27 100% 0% - 0% 1% 28 0 0 28 18 2 0 0 0 0 27 100% 0% - 0% 3% 116 137 0 253 98 250 2664 16 0 0 27 96% 0% - 20% 3% 35 160 0 195 184 1991 2300 0 0 0 27 98% 0% - 28% F840C*> ps -h Process statistics over 9.799 seconds... ID State Domain %CPU StackUsed %StackUsed Name 1 RR i 97% 276 6% idle_thread0 F840C*> ps -z 6% 25 114 0 139 14 58 1976 2240 0 0 27 98% 27% T 35% 3% 33 82 0 115 106 967 3916 16 0 0 27 95% 0% - 44% 4% 141 78 0 219 137 1005 4392 0 0 0 27 92% 0% - 42% 7% 105 74 0 179 53 700 5220 0 0 0 27 96% 0% - 51% 5% 56 148 0 204 317 1515 4292 16 0 0 27 97% 0% - 45% 6% 82 217 0 299 451 1408 4888 0 0 0 27 97% 0% - 39% 4% 164 163 0 327 86 2053 3628 0 0 0 27 98% 0% - 36% 1% 62 5 0 67 107 148 32 16 0 0 27 100% 0% - 1% F840C*> ps -h Process statistics over 7.586 seconds... ID State Domain %CPU StackUsed %StackUsed Name 1 RR i 98% 276 6% idle_thread0 0% 17 0 0 17 107 4 0 0 0 0 27 100% 0% - 0% 0% 20 0 0 20 1 1 0 0 0 0 27 100% 0% - 0% 1% 110 1 0 111 70 11 32 16 0 0 27 100% 0% - 1% 6% 39 165 0 204 107 78 1740 2128 0 0 27 98% 21% T 27% 8% 22 409 0 431 177 1691 5345 0 0 0 27 97% 0% - 35% F840C*> ps -z 4% 75 320 0 395 136 2526 2828 16 0 0 27 98% 0% - 20% 46% 104 404 0 508 179 432 0 0 0 0 27 100% 0% - 0% 94% 360 313 0 673 143 157 0 0 0 0 27 100% 0% - 0% 92% 34 311 0 345 131 86 40 16 0 0 27 100% 0% - 2% 48% 14 589 0 603 366 815 0 0 0 0 27 100% 0% - 0% 5% 209 633 0 842 205 1985 732 0 0 0 27 100% 0% - 15% F840C*> ps -h Process statistics over 5.988 seconds... ID State Domain %CPU StackUsed %StackUsed Name 1 RR i 51% 276 6% idle_thread0 17 BR k 18% 9060 27% FastEnet-10/100/e3c 8% 134 184 0 318 369 696 4463 16 0 0 27 97% 0% - 28% 1% 143 58 0 201 272 1871 156 0 0 0 27 97% 0% - 1%
F840C*> ps -h Process statistics over 386.288 seconds... ID State Domain %CPU StackUsed %StackUsed Name 1 RR i 84% 276 6% idle_thread0 17 BR k 5% 9060 27% FastEnet-10/100/e3c