One final thing I notice is that snapshots are taking sometime to delete which I know is an intensive operations in DOT

Example (at the bottom)

snapid status date ownblks release fsRev name

------ ------ ------------ ------- ------- ----- --------

61 complete Jun 16 15:49 334938 8.1 22331 2014-06-16_1548+0100_daily

60 complete Jun 16 15:48 765 8.1 22331 uk-su-ss007(1575077051)_SV_L_LUN_VOL0199-base.788

44 complete Jun 16 14:08 5415287 8.1 22331 2014-06-16_1408+0100_daily

37 complete Jun 16 10:45 11877364 8.1 22331 2014-06-16_1044+0100_daily

12 complete Jun 15 23:38 16471069 8.1 22331 2014-06-15_2338+0100_daily

250 complete Jun 15 23:36 6949 8.1 22331 2014-06-15_2334+0100_daily

90 complete Jun 15 23:30 16455 8.1 22331 uk-su-ss009(1575007753)_SV_M_LUN_VOL0199.120

58 complete Jun 15 23:29 1044 8.1 22331 2014-06-15_2328+0100_daily

226 complete Jun 15 23:23 40544 8.1 22331 2014-06-15_2322+0100_daily

216 complete Jun 15 15:10 4498948 8.1 22331 2014-06-15_1509+0100_daily

208 complete Jun 15 12:09 2605999 8.1 22331 2014-06-15_1208+0100_daily

196 complete Jun 15 09:24 2187941 8.1 22331 2014-06-15_0924+0100_daily

184 complete Jun 15 01:20 6696250 8.1 22331 2014-06-15_0120+0100_daily

176 complete Jun 15 01:18 5461 8.1 22331 2014-06-15_0117+0100_daily

161 complete Jun 15 01:14 32047 8.1 22331 2014-06-15_0113+0100_daily

148 complete Jun 15 01:11 56057 8.1 22331 2014-06-15_0111+0100_daily

124 complete Jun 14 15:09 4817241 8.1 22331 2014-06-14_1508+0100_daily

112 complete Jun 14 12:10 3349333 8.1 22331 2014-06-14_1210+0100_daily

94 complete Jun 14 10:03 3527119 8.1 22331 2014-06-14_1002+0100_daily

75 complete Jun 13 22:31 20042240 8.1 22331 2014-06-13_2229+0100_daily

69 complete Jun 13 22:27 4954 8.1 22331 2014-06-13_2226+0100_daily

52 complete Jun 13 21:36 343477 8.1 22331 2014-06-13_2135+0100_daily

41 complete Jun 13 20:54 8632101 8.1 22331 2014-06-13_2054+0100_daily

33 complete Jun 13 20:35 4298937 8.1 22331 2014-06-13_2034+0100_daily

19 complete Jun 13 20:32 142680 8.1 22331 2014-06-13_2031+0100_daily

1 complete Jun 13 11:09 6691044 8.1 22331 2014-06-13_1108+0100_daily

238 complete Jun 12 21:46 19191567 8.1 22331 2014-06-12_2145+0100_daily

224 complete Jun 12 21:05 9997690 8.1 22331 2014-06-12_2104+0100_daily

126 complete Jun 08 23:02 109922787 8.1 22331 2014-06-08_2301+0100_daily

130 complete Jun 08 22:58 4925 8.1 22331 2014-06-08_2257+0100_daily

179 complete Jun 08 22:52 21545 8.1 22331 2014-06-08_2251+0100_daily

167 complete Jun 08 22:44 41255 8.1 22331 2014-06-08_2243+0100_daily

85 complete Jun 08 01:05 11868850 8.1 22331 2014-06-08_0104+0100_daily

63 complete Jun 08 01:00 5647 8.1 22331 2014-06-08_0059+0100_daily

144 complete Jun 08 00:55 32549 8.1 22331 2014-06-08_0054+0100_daily

99 complete Jun 08 00:47 38928 8.1 22331 2014-06-08_0045+0100_daily

152 complete Jun 07 00:58 18053974 8.1 22331 2014-06-07_0057+0100_daily

228 complete Jun 07 00:54 4288 8.1 22331 2014-06-07_0052+0100_daily

54 complete Jun 07 00:46 61614 8.1 22331 2014-06-07_0045+0100_daily

15 complete Jun 07 00:39 73153 8.1 22331 2014-06-07_0038+0100_daily

139 complete Jun 06 01:38 34035727 8.1 22331 2014-06-06_0137+0100_daily

49 complete Jun 06 01:35 5386 8.1 22331 2014-06-06_0134+0100_daily

241 complete Jun 06 01:31 43381 8.1 22331 2014-06-06_0131+0100_daily

34 complete Jun 06 01:28 60157 8.1 22331 2014-06-06_0127+0100_daily

25 complete Jun 05 01:57 32400189 8.1 22331 2014-06-05_0156+0100_daily

77 complete Jun 05 01:55 4076 8.1 22331 2014-06-05_0154+0100_daily

200 complete Jun 05 01:51 75258 8.1 22331 2014-06-05_0151+0100_daily

132 complete Jun 05 01:48 85317 8.1 22331 2014-06-05_0147+0100_daily

9 complete Jun 04 00:00 39722504 8.1 22331 2014-06-03_2359+0100_daily

198 complete Jun 03 23:57 5421 8.1 22331 2014-06-03_2355+0100_daily

253 complete Jun 03 23:51 71954 8.1 22331 2014-06-03_2350+0100_daily

13 complete Jun 03 23:45 102485 8.1 22331 2014-06-03_2343+0100_daily

146 complete Jun 03 15:55 9263616 8.1 22331 2014-06-03_1555+0100_daily

128 complete Jun 03 15:14 4950630 8.1 22331 2014-06-03_1514+0100_daily

97 complete Jun 03 00:46 22248178 8.1 22331 2014-06-03_0045+0100_daily

166 complete Jun 03 00:43 4495 8.1 22331 2014-06-03_0042+0100_daily

104 complete Jun 03 00:37 68909 8.1 22331 2014-06-03_0036+0100_daily

70 complete Jun 03 00:31 145138 8.1 22331 2014-06-03_0030+0100_daily

254 complete Jun 02 21:20 35346082 8.1 22331 2014-06-02_2120+0100_daily

202 complete Jun 02 21:19 4114 8.1 22331 2014-06-02_2118+0100_daily

53 complete Jun 02 05:09 528753 8.1 22331 2014-06-02_0509+0100_daily

249 complete Jun 02 05:06 50924 8.1 22331 2014-06-02_0506+0100_daily

50 complete May 28 01:54 64891840 8.1 22331 2014-05-28_0154+0100_daily

5 complete May 28 01:51 6479 8.1 22331 2014-05-28_0150+0100_daily

151 complete May 28 01:49 70924 8.1 22331 2014-05-28_0148+0100_daily

28 complete May 28 01:46 79695 8.1 22331 2014-05-28_0146+0100_daily

129 complete May 27 15:55 10422515 8.1 22331 2014-05-27_1554+0100_daily

16 complete May 27 13:02 9773150 8.1 22331 2014-05-27_1301+0100_daily

157 complete May 27 10:27 10143739 8.1 22331 2014-05-27_1027+0100_daily

105 complete May 26 22:44 14249999 8.1 22331 2014-05-26_2243+0100_daily

252 complete May 26 22:41 7559 8.1 22331 2014-05-26_2240+0100_daily

160 complete May 26 22:37 27078 8.1 22331 2014-05-26_2236+0100_daily

23 complete May 26 22:31 44864 8.1 22331 2014-05-26_2230+0100_daily

232 complete May 26 15:37 4273421 8.1 22331 2014-05-26_1537+0100_daily

136 complete May 26 12:40 2211938 8.1 22331 2014-05-26_1240+0100_daily

91 complete May 26 09:47 3502863 8.1 22331 2014-05-26_0947+0100_daily

21 complete May 25 22:29 4984748 8.1 22331 2014-05-25_2228+0100_daily

4 complete May 25 22:26 6579 8.1 22331 2014-05-25_2225+0100_daily

133 complete May 25 22:21 19720 8.1 22331 2014-05-25_2220+0100_daily

51 complete May 25 22:17 29593 8.1 22331 2014-05-25_2216+0100_daily

194 complete May 25 15:38 2568742 8.1 22331 2014-05-25_1537+0100_daily

127 complete May 25 12:37 2007218 8.1 22331 2014-05-25_1237+0100_daily

30 complete May 25 09:45 1702320 8.1 22331 2014-05-25_0944+0100_daily

172 complete May 24 22:19 4880430 8.1 22331 2014-05-24_2218+0100_daily

143 complete May 24 22:16 4764 8.1 22331 2014-05-24_2215+0100_daily

171 complete May 24 22:08 26268 8.1 22331 2014-05-24_2207+0100_daily

115 complete May 24 22:02 37012 8.1 22331 2014-05-24_2201+0100_daily

3 complete May 24 15:37 3208176 8.1 22331 2014-05-24_1536+0100_daily

163 complete May 24 12:40 2092982 8.1 22331 2014-05-24_1239+0100_daily

246 complete May 24 09:54 2258149 8.1 22331 2014-05-24_0954+0100_daily

164 complete May 24 00:44 7307421 8.1 22331 2014-05-24_0042+0100_daily

173 complete May 24 00:39 8171 8.1 22331 2014-05-24_0037+0100_daily

65 complete May 24 00:34 63931 8.1 22331 2014-05-24_0034+0100_daily

188 complete May 24 00:27 58960 8.1 22331 2014-05-24_0026+0100_daily

153 complete May 23 16:02 8251951 8.1 22331 2014-05-23_1602+0100_daily

110 complete May 23 13:00 8945685 8.1 22331 2014-05-23_1259+0100_daily

87 complete May 23 10:32 9701026 8.1 22331 2014-05-23_1031+0100_daily

14 complete May 23 01:19 17123971 8.1 22331 2014-05-23_0118+0100_daily

235 complete May 23 01:17 10509 8.1 22331 2014-05-23_0117+0100_daily

111 complete May 23 01:15 13058 8.1 22331 2014-05-23_0115+0100_daily

46 complete May 23 01:13 239438 8.1 22331 2014-05-23_0112+0100_daily

230 complete May 22 02:20 39300529 8.1 22331 2014-05-22_0220+0100_daily

89 complete May 22 02:18 5642 8.1 22331 2014-05-22_0217+0100_daily

32 complete May 22 02:14 41995 8.1 22331 2014-05-22_0213+0100_daily

22 complete May 22 02:11 174295 8.1 22331 2014-05-22_0210+0100_daily

93 complete May 21 01:36 42877364 8.1 22331 2014-05-21_0136+0100_daily

255 complete May 21 01:33 7137 8.1 22331 2014-05-21_0132+0100_daily

165 complete May 21 01:29 72219 8.1 22331 2014-05-21_0128+0100_daily

95 complete May 21 01:25 97763 8.1 22331 2014-05-21_0124+0100_daily

212 complete May 20 02:24 36695419 8.1 22331 2014-05-20_0223+0100_daily

177 complete May 20 02:22 2988 8.1 22331 2014-05-20_0222+0100_daily

131 complete May 20 02:18 42998 8.1 22331 2014-05-20_0218+0100_daily

227 complete May 20 02:15 78827 8.1 22331 2014-05-20_0214+0100_daily

155 complete May 18 23:07 39970911 8.1 22331 2014-05-18_2306+0100_daily

123 complete May 18 23:04 5246 8.1 22331 2014-05-18_2302+0100_daily

102 complete May 18 22:57 16970 8.1 22331 2014-05-18_2257+0100_daily

81 complete May 18 22:53 31928 8.1 22331 2014-05-18_2252+0100_daily

251 complete May 17 23:34 14464864 8.1 22331 2014-05-17_2333+0100_daily

147 complete May 17 23:30 8776 8.1 22331 2014-05-17_2328+0100_daily

45 complete May 17 23:25 16371 8.1 22331 2014-05-17_2324+0100_daily

223 complete May 17 23:19 36615 8.1 22331 2014-05-17_2318+0100_daily

98 complete May 16 23:21 20601283 8.1 22331 2014-05-16_2320+0100_daily

66 complete May 16 23:17 6112 8.1 22331 2014-05-16_2316+0100_daily

27 complete May 16 23:10 24164 8.1 22331 2014-05-16_2310+0100_daily

17 complete May 16 23:04 102933 8.1 22331 2014-05-16_2302+0100_daily

225 complete May 15 23:15 34033701 8.1 22331 2014-05-15_2314+0100_daily

187 complete May 15 23:12 5472 8.1 22331 2014-05-15_2311+0100_daily

96 complete May 15 23:07 22786 8.1 22331 2014-05-15_2306+0100_daily

10 complete May 15 23:01 71324 8.1 22331 2014-05-15_2300+0100_daily

191 complete May 14 23:28 33533416 8.1 22331 2014-05-14_2327+0100_daily

80 complete May 14 23:23 6759 8.1 22331 2014-05-14_2322+0100_daily

231 complete May 14 23:17 25124 8.1 22331 2014-05-14_2316+0100_daily

183 complete May 14 23:10 64807 8.1 22331 2014-05-14_2309+0100_daily

118 complete May 13 23:26 32621854 8.1 22331 2014-05-13_2325+0100_daily

203 complete May 13 23:21 5396 8.1 22331 2014-05-13_2320+0100_daily

154 complete May 13 23:14 20931 8.1 22331 2014-05-13_2313+0100_daily

134 complete May 13 23:08 51678 8.1 22331 2014-05-13_2307+0100_daily

101 deleting May 13 01:43 33375746 134.0 (319143/319143 remaining)

67 deleting May 13 01:41 3210 134.0 (319143/319143 remaining)

245 deleting May 13 01:37 0 67.0 (143401/319143 remaining)

From: Jordan Slingerland [mailto:Jordan.Slingerland@independenthealth.com]
Sent: 16 June 2014 21:45
To: Burchell, Will (ITSD); Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

I suppose it could depend but If I had to throw out a number, 10m would probably be a good start in most situations.

--Jordan

From: Will.Burchell@skanska.co.uk [mailto:Will.Burchell@skanska.co.uk]
Sent: Monday, June 16, 2014 4:38 PM
To: Jordan Slingerland; marcus.nilsson@atea.se; Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

Thanks

Happy to run wafltop and dump the ouput for comment. Just the standard volume and process?

How long is it a good idea to leave it running to get a useful output

William

From: Jordan Slingerland [mailto:Jordan.Slingerland@independenthealth.com]
Sent: 16 June 2014 21:24
To: Burchell, Will (ITSD); marcus.nilsson@atea.se; Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

I am not actually sure that is an issue...but I was told it is by IBM n-series support. I do have an open case currently escalated from IBM to Netapp regarding the same issue with stale metadata. IBM told me to run sis start –s once, and then sis start manually 2x on each volume.(they specifically said sis start (no switches) volume ) had to be run 2x on each volume and a scheduled run would not suffice) Still have some over 100% stale on several volumes.

Maybe it was just a way to keep me busy for a week. “um yeah, go dedup all your volumes 3 times and come back if it does’t help”

More info here, but it sounds like you already got that.

https://library.netapp.com/ecmdocs/ECMP1368838/html/GUID-5B6B2A2E-FAFD-4A92-B8FB-01E7CD876FE3.html

Perhaps post your wafltop too and someone might be able to point something out in that.

--Jordan

From: Will.Burchell@skanska.co.uk [mailto:Will.Burchell@skanska.co.uk]
Sent: Monday, June 16, 2014 3:56 PM
To: Jordan Slingerland; marcus.nilsson@atea.se; Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

No deduplications are running

So I have run sis status –l and can confirm the following SIS jobs and their stale fingerprints. This looks pretty bad. What are we doing wrong here?

We have upgraded from various versions of ONTAP last year and believe we ran into SIS issue but thought they had been cleared by running the sis start –S command to clean them out

We are on 8.1.3P2 and came from 8.0.x into many versions of 8.1.x over the last 18 months

Will

79%

14%

52%

130%

108%

126%

117%

181%

112%

121%

81%

25%

26%

From: Jordan Slingerland [mailto:Jordan.Slingerland@independenthealth.com]
Sent: 16 June 2014 20:47
To: Marcus Nilsson; Burchell, Will (ITSD); Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

It sounds like you ruled out the obvious, but I will say it anyway. but no deduplciations running, right?

And not so obvious, If none running, look at sis status –l and check if any of the volumes are over 20% in the Stale Fingerprints: column.

--Jordan

From: Marcus Nilsson [mailto:marcus.nilsson@atea.se]
Sent: Monday, June 16, 2014 3:44 PM
To: Will.Burchell@skanska.co.uk; Jordan Slingerland; Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

Hi,

Might be worth checking out the article at this link http://www.jk-47.com/2014/02/attack-of-old-bugs-netapp-high-cpu/

We ran into this exact issue after upgrading a system from 8.0.3P2 to 8.1.4P1. A process looping in wafl scan blk_reclaim.

BR Marcus

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Will.Burchell@skanska.co.uk
Sent: den 16 juni 2014 21:19
To: Jordan.Slingerland@independenthealth.com; Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

I am checking all cpus and they are pretty busy

We are in the UK so it’s out of hours (and our nightly process are mostly stopped right now)

We have the issue I mentioned where our exchange LUNs are on the same aggregate together and we have a high IO workload with 6000 mailboxes.

This is the sysstat –M 1 right now as an example. It seems high considering there is no de-dupe and only a single snapmirror running (to do a vol move for our exchange separation problem)

Any other thoughts and I am most interested

William

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

92% 76% 56% 32% 68% 67% 73% 75% 59% 47% 0% 0% 22% 26% 0% 12% 105%( 65%) 0% 5% 0% 34% 9% 13% 9152 0%

98% 90% 80% 60% 85% 87% 89% 88% 75% 31% 0% 0% 26% 42% 0% 15% 122%( 72%) 21% 11% 0% 52% 9% 9% 5637 50%

99% 92% 80% 55% 84% 86% 90% 91% 69% 31% 0% 0% 29% 58% 0% 16% 103%( 66%) 3% 11% 0% 67% 9% 8% 5962 100%

98% 91% 79% 54% 84% 84% 88% 90% 72% 45% 0% 0% 27% 45% 0% 9% 122%( 78%) 0% 12% 0% 54% 8% 10% 7222 100%

99% 94% 88% 67% 89% 91% 94% 94% 77% 25% 0% 0% 28% 63% 0% 21% 100%( 65%) 25% 9% 0% 71% 8% 7% 4452 100%

97% 91% 79% 52% 83% 84% 89% 91% 67% 39% 0% 0% 30% 51% 0% 8% 113%( 73%) 0% 10% 0% 62% 8% 9% 8253 100%

98% 87% 71% 44% 78% 79% 84% 83% 67% 46% 0% 0% 25% 33% 0% 14% 121%( 74%) 0% 9% 0% 42% 12% 11% 9237 66%

97% 93% 86% 65% 88% 87% 92% 93% 80% 29% 0% 0% 27% 50% 0% 22% 116%( 69%) 24% 9% 0% 59% 9% 8% 5213 63%

97% 85% 69% 42% 76% 77% 83% 85% 60% 37% 0% 0% 28% 42% 0% 9% 109%( 69%) 1% 10% 0% 48% 11% 10% 6795 100%

98% 91% 77% 50% 82% 83% 88% 91% 66% 39% 0% 0% 30% 50% 0% 8% 116%( 73%) 0% 10% 0% 58% 7% 10% 6993 100%

98% 92% 82% 62% 86% 85% 90% 91% 78% 29% 0% 0% 28% 51% 0% 20% 108%( 69%) 21% 14% 0% 59% 6% 8% 5308 90%

100% 97% 91% 65% 90% 92% 94% 96% 80% 30% 0% 0% 30% 59% 0% 16% 120%( 76%) 3% 20% 0% 68% 9% 7% 5593 100%

98% 85% 70% 47% 78% 76% 82% 81% 71% 33% 0% 0% 26% 41% 0% 16% 110%( 70%) 4% 12% 0% 48% 10% 10% 5907 79%

100% 98% 89% 61% 89% 91% 94% 96% 75% 28% 0% 0% 32% 62% 0% 20% 98%( 65%) 17% 10% 0% 73% 8% 7% 5290 100%

98% 91% 77% 50% 82% 80% 85% 89% 72% 33% 0% 0% 30% 48% 0% 21% 108%( 64%) 0% 12% 0% 59% 6% 9% 6047 100%

99% 91% 75% 49% 82% 80% 84% 85% 77% 36% 0% 0% 26% 29% 0% 15% 144%( 80%) 1% 12% 0% 44% 10% 10% 6412 67%

100% 95% 88% 68% 90% 88% 94% 97% 80% 26% 0% 0% 29% 59% 0% 26% 100%( 66%) 23% 16% 0% 65% 8% 7% 4602 100%

98% 87% 74% 48% 79% 78% 86% 90% 63% 30% 0% 0% 29% 52% 0% 9% 105%( 68%) 0% 14% 0% 60% 9% 8% 5533 100%

98% 88% 77% 58% 83% 81% 87% 90% 73% 30% 0% 0% 27% 47% 0% 19% 106%( 66%) 21% 10% 0% 54% 9% 8% 5691 98%

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

97% 86% 70% 43% 77% 77% 84% 87% 61% 39% 0% 0% 27% 39% 0% 7% 116%( 73%) 0% 11% 0% 49% 11% 10% 7526 100%

97% 86% 70% 44% 78% 80% 85% 87% 61% 34% 0% 0% 28% 44% 0% 9% 108%( 68%) 0% 14% 0% 53% 11% 13% 6308 100%

98% 87% 77% 59% 82% 80% 86% 88% 76% 28% 0% 0% 24% 44% 0% 23% 106%( 66%) 21% 14% 0% 53% 9% 8% 5200 82%

100% 96% 86% 57% 87% 88% 92% 95% 73% 30% 0% 0% 30% 56% 0% 18% 111%( 69%) 3% 18% 0% 68% 6% 8% 5163 100%

98% 90% 78% 55% 83% 82% 88% 91% 69% 32% 0% 0% 28% 44% 0% 11% 119%( 74%) 6% 19% 0% 54% 9% 9% 6148 99%

100% 97% 89% 64% 89% 91% 93% 96% 75% 34% 0% 0% 30% 62% 0% 22% 99%( 65%) 17% 9% 0% 70% 6% 8% 6496 100%

From: Jordan Slingerland [mailto:Jordan.Slingerland@independenthealth.com]
Sent: 16 June 2014 20:14
To: Burchell, Will (ITSD); Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

Even if it is 10k ops after 5 minutes...that is only 33 ops per second. I doubt 33 unaligned ops per second is your cpu issue.

Maybe you can fix that one top talker just to show support that is not the issue? …depending how critical that 1 system is that may or may not be worth fighting over support with.

Now, on to the cpu issue. Are using “sysstat –m 1” to look at all cpus and not only the “ANY” cpu metric right?

If you do , for example, “sysstat –x 1” you are looking at the % of time that ANY of your cpus are busy. Seems to me this metric is nearly completely useless.

--Jordan

From: Will.Burchell@skanska.co.uk [mailto:Will.Burchell@skanska.co.uk]
Sent: Monday, June 16, 2014 3:07 PM
To: Jordan Slingerland; Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

Thanks

I reset with the –z switch

I then run –d again a 5 minutes later. Many of the counters are in the 10’s so I am happy with this. However 1 server is in the thousands already. This is a windows 2000 server (don’t ask please!) which has a misaligned C drive but I have used the “functional aligned” datastore in VSC to get around this. I assume nfsstat –d won’t understand that hence the counters in the thousands

William

From: Jordan Slingerland [mailto:Jordan.Slingerland@independenthealth.com]
Sent: 16 June 2014 19:57
To: Burchell, Will (ITSD); Toasters@teaparty.net
Subject: RE: High CPU VM misalignment confusion

First off, make sure the values in nfsstat –d are actually incrementing significantly by running nfsstat –z to clear the counters and then wait a while and looking at nfsstat –d again.

You may find that you are only doing a handful of unaligned ops and not hundreds or thousands per second.

--Jordan

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Will.Burchell@skanska.co.uk
Sent: Monday, June 16, 2014 2:50 PM
To: Toasters@teaparty.net
Subject: High CPU VM misalignment confusion

Hello. I am hoping you can guide me in the right direction

We have been experiencing very high CPU load on a 7-mode HA pair of 3270 controllers run 8.1.3P2

We have worked with netapp support on these issues and they note our workload is very high on one controller (where we run our VMware setup from)

We also have so called “bad practice” where we are running our exchange ISCSI LUNs on SATA with logs and dbs on the same aggregate (currently separating this out as I type)

I have been told by support we have VMDK misalignment, however I spent a long time a few months ago resolving this firstly by using the VSC tool to confirm the problem and then fixing it with a combination of MBRALIGN and VMware converter as a V2V process

The support guy tells me he seems misalignment when he runs nfsstat –d but MBRSCAN shows these are aligned. What is going on here?

Trying to reduce our CPU and IO burden but getting conflicting information.

Finally I think we should look to upgrade to 8.1.4P2 to remove some bugs? We would consider 8.2.x but I don’t think we can as we run Exchange 2010 (using SME 6.x etc)

Thanks in advance

William