JFTR: Upgraded to 8.1.3P2 yesterday, everything good again – dedupes are no longer causing performance issues, so this was the solution.

 

Bye,

 

Alexander Griesser

System-Administrator

 

ANEXIA Internetdienstleistungs GmbH

 

Telefon: +43-5-0556-320

Telefax: +43-5-0556-500

 

E-Mail: ag@anexia.at

Web: http://www.anexia.at

 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

 

Von: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag von Aaron Lewis
Gesendet: Samstag, 19. Oktober 2013 02:22
An: toasters@teaparty.net
Betreff: RE: AW: Dedup killing performance (again)

 

I’ll have to agree as well, we’re recommending all of our customers go to either 8.1.3 or 8.2P3.  We’ve had several people suffer from the dedupe fingerprinting issue in 8.1.2 and now there’s a serious data corruption bug (#723354) in 8.2 pre-P3.

 

Bug description:

 

 

o   burt 723354 “Illegal container vbn panic while loading a BLKR MOVED buf”; SW, dense, sev-2. A problem in all current 8.2 releases (8.2RC1, 8.2GA, 8.2P1, 8.2P2, and any D-patch derivatives) which may result in WAFL inconsistencies on systems running either 7-Mode or clustered Data ONTAP if they have dedupe-enabled volumes and there is repeated overwriting of 4KB blocks while a deduplication process is running against that volume. 
A is included in Data ONTAP 8.2P3 and existing customers using dedupe on 8.2 are advised to disable dedupe now to avoid further exposure, and then upgrade to 8.2P3 as soon as is operationally feasable (at which time dedupe can be re-enabled). It is also recommended that customers using dedupe on a pre-8.2 release of Data ONTAP that are considering upgrading to Data ONTAP 8.2 upgrade to 8.2P3 (or later) to avoid this issue.

 

 

 

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alexander Griesser
Sent: Friday, October 18, 2013 5:13 PM
To: Fred Grieco; Mark Allen
Cc: Peter-Paul Witta; toasters@teaparty.net
Subject: AW: AW: Dedup killing performance (again)

 

Alright, I’m almost convinced to go for 8.1.3 now – will read through the changelogs tomorrow morning; right now, everything is fine again, fingerprint recovery has been finished a few minutes after my last mail regarding this issue and I’m now deleting some old snapshots to regain that space which should be fine for now.

 

I’ve disabled any future sis schedules for now until I know how I want to proceed with that situation.

 

Thanks for all your fast help as always J

 

Bye,

 

Alexander Griesser

System-Administrator

 

ANEXIA Internetdienstleistungs GmbH

 

Telefon: +43-5-0556-320

Telefax: +43-5-0556-500

 

E-Mail: ag@anexia.at

Web: http://www.anexia.at

 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

 

Von: Fred Grieco [mailto:fredgrieco@yahoo.com]
Gesendet: Samstag, 19. Oktober 2013 02:11
An: Alexander Griesser; Mark Allen
Cc: Peter-Paul Witta;
toasters@teaparty.net
Betreff: Re: AW: Dedup killing performance (again)

 

I have 8.1.3p1 on a 3140 and a 3270 (7 mode) with no dedupe performance issues.

Sent from Yahoo! Mail on Android

 


From: Mark Allen <mmmjka@gmail.com>;
To: Alexander Griesser <
ag@anexia.at>;
Cc: Peter-Paul Witta <
paul.witta@cubit.at>; <toasters@teaparty.net>;
Subject: Re: AW: Dedup killing performance (again)
Sent: Sat, Oct 19, 2013 12:00:06 AM

 

I would agree!! I upgraded to 8.1.3p2 on several arrays over the past month and there hasn&apos;t been any fall out.

-Mark

On Oct 18, 2013 4:39 PM, "Alexander Griesser" <ag@anexia.at> wrote:

Is it stable? I mean 8.1.3 is just in GA for now…

 

Alexander Griesser

System-Administrator

 

ANEXIA Internetdienstleistungs GmbH

 

Telefon: +43-5-0556-320

Telefax: +43-5-0556-500

 

E-Mail: ag@anexia.at

Web: http://www.anexia.at

 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

 

Von: Peter-Paul Witta [mailto:paul.witta@CUBiT.at]
Gesendet: Samstag, 19. Oktober 2013 00:30
An: Alexander Griesser; Sebastian Goetze
Cc:
toasters@teaparty.net
Betreff: Re: Dedup killing performance (again)

 

I&apos;d recommend 813p2 immediately...

lg
paul witta


Am 19.10.13 00:19, schrieb Alexander Griesser:

Yah, just found it while googling I guess, must be bug id #657692 :-/

 

Anyhow, I’m now stuck with this:

 

> sis status

Path                           State      Status     Progress

/vol/CIFS_SHARES               Enabled    Active     Fingerprint is being recovered from aggregate (6% complete)

 

It’s doing that since quite some time now, although it now actually jumped from 0% to 6%, so it at least is still alive.

 

This bug is first fixed in  8.1.2P4 as it seems, so I might need to upgrade ASAP; for now, I’ll check if it at least finishes this process now, my volume is filling up quickly, but there’s still some aggregate space left so I’ll just increase the volume size before it goes full.

I’ll go above the magic 80% fill of an aggregate then, but better that then having the volume go full.

 

I’ll see how it goes and if it troubles me too much, I hope there is a way to abort sis in that state.

 

Alexander Griesser

System-Administrator

 

ANEXIA Internetdienstleistungs GmbH

 

Telefon: +43-5-0556-320

Telefax: +43-5-0556-500

 

E-Mail: ag@anexia.at

Web: http://www.anexia.at

 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

 

Von: Sebastian Goetze [mailto:spgoetze@gmail.com]
Gesendet: Samstag, 19. Oktober 2013 00:16
An: Alexander Griesser
Betreff: Re: Dedup killing performance (again)

 

Isn&apos;t that the Dedupe bug again, that was fixed in 8.1.3? 
(since like 8.1.2P3 or P4)...

There was a thread about it some time ago...

Anyway, I&apos;m off to bed, just wanted to point you in the right direction.

BFN

On 10/18/2013 11:52 PM, Alexander Griesser wrote:

Hi there,

 

I am just trying to dedupe some volumes on one of our filers and the first volume (2.4TB data, CIFS share) deduplicates since about 4 hours now.

Sysstat –x 1 shows:

 

CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s

                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out

58%    115      0      0     116    2239   3635  281593     32       0      0     0s    98%    0%  -   100%       0      0      1       0      0       0      0

63%    162      0      0     169    1682   4149  310829      0       0      0     0s    99%    0%  -   100%       7      0      0       0      0       0      0

50%    381      0      0     384   11793  11325  182100      0       0      0     0s    96%    0%  -   100%       0      0      3       0      0       0      0

55%    412      0      0     416    6487   6040  223036     24       0      0     0s    97%    0%  -   100%       0      0      4       0      0      37      0

51%     19      0      0      19     185      9  239388      0       0      0     2     98%    0%  -   100%       0      0      0       0      0       0      0

51%     77      0      0      80     348    906  273880      8       0      0     2     99%    0%  -   100%       0      0      3       0      0       0      0

60%     48      0      0      49     174    410  196179    271       0      0     2     98%   35%  Tn  100%       0      0      1       0      0       0      0

45%    102      0      0     110     337    337  144387  21222       0      0     2     96%  100%  :f  100%       8      0      0       0      0       0      0

71%    143      0      0     149     258   2814  109728  24268       0      0     2     85%  100%  :f   86%       0      0      6       0      0      25      0

53%     64      0      0      65     203   1368  236620  14152       0      0     2     98%  100%  :f  100%       0      0      1       0      0       0      0

49%     96      0      0      96     319   1317  185552  19748       0      0     2     96%  100%  :f  100%       0      0      0       0      0       0      0

35%    109      0      0     117     381   1962   62613  18010       0      0     2     88%  100%  :f  100%       5      0      3       0      0       0      0

55%     38      0      0      39     159   1168  239408   1980       0      0     2     97%  100%  :f  100%       0      0      1       0      0       0      0

57%    177      0      0     179     308   6692  136568    520       0      0     2     92%   82%  :    99%       0      0      2       0      0      37      0

39%    256      0      0     271     403  12647  117284      0       0      0     2     93%    0%  -    97%      12      0      3       0      0       0      0

40%    232      0      0     473     808   7506  162368     32       0      0     2     96%    0%  -   100%     240      0      1       0      0       0      0

38%     86      0      0      86     246   3413  160092      0       0      0     2     96%    0%  -   100%       0      0      0       0      0       0      0

47%    113      0      0     117     314   1738  199516      0       0      0     2     96%    0%  -   100%       1      0      3       0      0       0      0

59%    116      0      0     120     476   2751  288048     24       0      0     0s    99%    0%  -   100%       0      0      4       0      0      24      0

45%    230      0      0     230    2879   6547  185787    120       0      0     0s    98%    0%  -   100%       0      0      0       0      0       0      0

 

> version

NetApp Release 8.1.2P2 7-Mode: Wed Feb 20 19:58:27 PST 2013

 

I’m aware that deduplication _SHOULD_ be a background process hugging up as much resources as it can but when host i/o is needed, it will be preferred over deduplication.

There are no other tasks running at the same time (snapvault, etc.), and the virtual machinse on this system are _VERY_ slow – f.ex. logging in via RDP on one of the virtual machines now took > 1 minute for the login window to appear.

 

This is a FAS2240 with 24x600 SAS 10k and basically my question now is: Is there any way to tell a-sis to really run with lower priority or is this a bug in the OnTap version I’m running? I’ve used a-sis a lot in the past and never had performance issues like this when it was running…

 

I’m thankful for any ideas how to track down this issue or improve performance while a-sis is running.

 

Thanks,

 

Alexander Griesser

System-Administrator

 

ANEXIA Internetdienstleistungs GmbH

 

Telefon: +43-5-0556-320

Telefax: +43-5-0556-500

 

E-Mail: ag@anexia.at

Web: http://www.anexia.at

 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt

Geschäftsführer: Alexander Windbichler

Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

 

 

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters

 

 

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters

 

-- 
kind regards,                 +43-664-4542287
Ing. Peter-Paul Witta         +43-1-7189880-0
Chief Technology Officer        www.CUBiT.at


_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters