I don't know that error, but you can see the constituents by: vol show -vserver vserver_name -is-constituent true
-jeff
From: Toasters toasters-bounces@teaparty.net on behalf of "Rue, Randy" randyrue@gmail.com Date: Wednesday, May 26, 2021 at 11:28 AM To: Toasters toasters@teaparty.net Subject: [SUSPECTED SPAM]WAFL error on a flexgroup?
External Email - Use Caution
Hello All,
Seeing this error repeatedly in the logs:
wafl.vol.fsp.full:error]: volume scharp_systems__0002@vserver:51aa02de-9c6e-11eb-9f37-d039ea257af4: insufficient space in FSP wafl_remote_reserve to satisfy a request of 0 holes and 12 overwrites.
An online search is giving me nothing meaningful, not even NetApp specific results.
Note that this is a constituent volume for a larger flexgroup (which are new to us).
Is the constituent volume full? If so, is there some reason data isn't balanced automatically across the constituents of a flexgroup?
Hope to hear from you,
Randy in Seatlle
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
********************************************************** Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues
sure enough, that constituent volume is completely full:
scarab::> vol show -vserver scharpdata -is-constituent true Vserver Volume Aggregate State Type Size Available Used% --------- ------------ ------------ ---------- ---- ---------- ---------- ----- scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB 0B 100%
now what?
On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote:
vol show -vserver vserver_name -is-constituent true
OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell?
On 5/26/2021 8:47 AM, Rue, Randy wrote:
sure enough, that constituent volume is completely full:
scarab::> vol show -vserver scharpdata -is-constituent true Vserver Volume Aggregate State Type Size Available Used%
scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB 0B 100%
now what?
On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote:
vol show -vserver vserver_name -is-constituent true
What ONTAP version are you on?
Have you reviewed the FG TR? We state in there that we don't completely balance evenly across the FG for a variety of reasons. And there's no post placement data rebalance yet.
https://www.netapp.com/us/media/tr-4571.pdf
-----Original Message----- From: Toasters toasters-bounces@teaparty.net On Behalf Of Rue, Randy Sent: Wednesday, May 26, 2021 11:58 AM To: Toasters toasters@teaparty.net Subject: Re: WAFL error on a flexgroup?
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell?
On 5/26/2021 8:47 AM, Rue, Randy wrote:
sure enough, that constituent volume is completely full:
scarab::> vol show -vserver scharpdata -is-constituent true Vserver Volume Aggregate State Type Size Available Used%
scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB 0B 100%
now what?
On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote:
vol show -vserver vserver_name -is-constituent true
_______________________________________________ Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
I believe this might help:
Elastic sizing was introduced in Ontap 9.6. This article details the issue: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP...
This should be turned on by default but can be turned off via: node run nodename flexgroup set elastic-sizing=false
On Wed, May 26, 2021 at 12:01 PM Rue, Randy randyrue@gmail.com wrote:
OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell?
On 5/26/2021 8:47 AM, Rue, Randy wrote:
sure enough, that constituent volume is completely full:
scarab::> vol show -vserver scharpdata -is-constituent true Vserver Volume Aggregate State Type Size Available Used%
scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB 0B 100%
now what?
On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote:
vol show -vserver vserver_name -is-constituent true
Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
I’d actually recommend 9.8 and proactive resizing instead.
Elastic sizing is reactive and incurs a latency penalty when used. And it can’t be used with vol autosize.
Proactive resizing starts resizing when we get to 60% used and maintains a relatively even free space buffer and can be used with autosize.
From: Toasters toasters-bounces@teaparty.net On Behalf Of Douglas Siggins Sent: Wednesday, May 26, 2021 12:15 PM To: Rue, Randy randyrue@gmail.com Cc: Toasters toasters@teaparty.net Subject: Re: WAFL error on a flexgroup?
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I believe this might help:
Elastic sizing was introduced in Ontap 9.6. This article details the issue: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP...
This should be turned on by default but can be turned off via: node run nodename flexgroup set elastic-sizing=false
On Wed, May 26, 2021 at 12:01 PM Rue, Randy <randyrue@gmail.commailto:randyrue@gmail.com> wrote: OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell?
On 5/26/2021 8:47 AM, Rue, Randy wrote:
sure enough, that constituent volume is completely full:
scarab::> vol show -vserver scharpdata -is-constituent true Vserver Volume Aggregate State Type Size Available Used%
scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB 0B 100%
now what?
On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote:
vol show -vserver vserver_name -is-constituent true
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
We're at 9.8P1. I'll dig into the below link.
On 5/26/2021 9:23 AM, Parisi, Justin wrote:
I’d actually recommend 9.8 and proactive resizing instead.
Elastic sizing is reactive and incurs a latency penalty when used. And it can’t be used with vol autosize.
Proactive resizing starts resizing when we get to 60% used and maintains a relatively even free space buffer and can be used with autosize.
*From:* Toasters toasters-bounces@teaparty.net *On Behalf Of *Douglas Siggins *Sent:* Wednesday, May 26, 2021 12:15 PM *To:* Rue, Randy randyrue@gmail.com *Cc:* Toasters toasters@teaparty.net *Subject:* Re: WAFL error on a flexgroup?
*NetApp Security WARNING*: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I believe this might help:
Elastic sizing was introduced in Ontap 9.6. This article details the issue:
https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP... https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/Elastic_Sizing_with_FlexGroup_Volumes
This should be turned on by default but can be turned off via:
node run nodename flexgroup set elastic-sizing=false
On Wed, May 26, 2021 at 12:01 PM Rue, Randy <randyrue@gmail.com mailto:randyrue@gmail.com> wrote:
OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell? On 5/26/2021 8:47 AM, Rue, Randy wrote: > sure enough, that constituent volume is completely full: > > scarab::> vol show -vserver scharpdata -is-constituent true > Vserver Volume Aggregate State Type Size > Available Used% > --------- ------------ ------------ ---------- ---- ---------- > ---------- ----- > scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB > 0B 100% > > > now what? > > > On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote: >> vol show -vserver vserver_name -is-constituent true _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> https://www.teaparty.net/mailman/listinfo/toasters <https://www.teaparty.net/mailman/listinfo/toasters>
OK, after reading the below it sounds like elastic sizing should already be on and should have prevented my full constituent volume. Why didn't it?
What's proactive resizing? Where can I find out more about it?
On 5/26/2021 9:23 AM, Parisi, Justin wrote:
I’d actually recommend 9.8 and proactive resizing instead.
Elastic sizing is reactive and incurs a latency penalty when used. And it can’t be used with vol autosize.
Proactive resizing starts resizing when we get to 60% used and maintains a relatively even free space buffer and can be used with autosize.
*From:* Toasters toasters-bounces@teaparty.net *On Behalf Of *Douglas Siggins *Sent:* Wednesday, May 26, 2021 12:15 PM *To:* Rue, Randy randyrue@gmail.com *Cc:* Toasters toasters@teaparty.net *Subject:* Re: WAFL error on a flexgroup?
*NetApp Security WARNING*: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I believe this might help:
Elastic sizing was introduced in Ontap 9.6. This article details the issue:
https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP... https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/Elastic_Sizing_with_FlexGroup_Volumes
This should be turned on by default but can be turned off via:
node run nodename flexgroup set elastic-sizing=false
On Wed, May 26, 2021 at 12:01 PM Rue, Randy <randyrue@gmail.com mailto:randyrue@gmail.com> wrote:
OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell? On 5/26/2021 8:47 AM, Rue, Randy wrote: > sure enough, that constituent volume is completely full: > > scarab::> vol show -vserver scharpdata -is-constituent true > Vserver Volume Aggregate State Type Size > Available Used% > --------- ------------ ------------ ---------- ---- ---------- > ---------- ----- > scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB > 0B 100% > > > now what? > > > On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote: >> vol show -vserver vserver_name -is-constituent true _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> https://www.teaparty.net/mailman/listinfo/toasters <https://www.teaparty.net/mailman/listinfo/toasters>
TR-4571
https://www.netapp.com/us/media/tr-4571.pdf
From: Rue, Randy randyrue@gmail.com Sent: Wednesday, May 26, 2021 2:43 PM To: Parisi, Justin Justin.Parisi@netapp.com; Douglas Siggins siggins@gmail.com Cc: Toasters toasters@teaparty.net Subject: Re: WAFL error on a flexgroup?
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
OK, after reading the below it sounds like elastic sizing should already be on and should have prevented my full constituent volume. Why didn't it?
What's proactive resizing? Where can I find out more about it?
On 5/26/2021 9:23 AM, Parisi, Justin wrote: I’d actually recommend 9.8 and proactive resizing instead.
Elastic sizing is reactive and incurs a latency penalty when used. And it can’t be used with vol autosize.
Proactive resizing starts resizing when we get to 60% used and maintains a relatively even free space buffer and can be used with autosize.
From: Toasters toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net On Behalf Of Douglas Siggins Sent: Wednesday, May 26, 2021 12:15 PM To: Rue, Randy randyrue@gmail.commailto:randyrue@gmail.com Cc: Toasters toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: WAFL error on a flexgroup?
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I believe this might help:
Elastic sizing was introduced in Ontap 9.6. This article details the issue: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP...
This should be turned on by default but can be turned off via: node run nodename flexgroup set elastic-sizing=false
On Wed, May 26, 2021 at 12:01 PM Rue, Randy <randyrue@gmail.commailto:randyrue@gmail.com> wrote: OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell?
On 5/26/2021 8:47 AM, Rue, Randy wrote:
sure enough, that constituent volume is completely full:
scarab::> vol show -vserver scharpdata -is-constituent true Vserver Volume Aggregate State Type Size Available Used%
scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB 0B 100%
now what?
On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote:
vol show -vserver vserver_name -is-constituent true
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
Thank you.
I'm not finding any mention of turning this feature on or off, is the default? If so, we're at 9.8P1 and should be running it.
On 5/26/2021 11:44 AM, Parisi, Justin wrote:
TR-4571
https://www.netapp.com/us/media/tr-4571.pdf https://www.netapp.com/us/media/tr-4571.pdf
*From:* Rue, Randy randyrue@gmail.com *Sent:* Wednesday, May 26, 2021 2:43 PM *To:* Parisi, Justin Justin.Parisi@netapp.com; Douglas Siggins siggins@gmail.com *Cc:* Toasters toasters@teaparty.net *Subject:* Re: WAFL error on a flexgroup?
*NetApp Security WARNING*: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
OK, after reading the below it sounds like elastic sizing should already be on and should have prevented my full constituent volume. Why didn't it?
What's proactive resizing? Where can I find out more about it?
On 5/26/2021 9:23 AM, Parisi, Justin wrote:
I’d actually recommend 9.8 and proactive resizing instead. Elastic sizing is reactive and incurs a latency penalty when used. And it can’t be used with vol autosize. Proactive resizing starts resizing when we get to 60% used and maintains a relatively even free space buffer and can be used with autosize. *From:* Toasters <toasters-bounces@teaparty.net> <mailto:toasters-bounces@teaparty.net> *On Behalf Of *Douglas Siggins *Sent:* Wednesday, May 26, 2021 12:15 PM *To:* Rue, Randy <randyrue@gmail.com> <mailto:randyrue@gmail.com> *Cc:* Toasters <toasters@teaparty.net> <mailto:toasters@teaparty.net> *Subject:* Re: WAFL error on a flexgroup? *NetApp Security WARNING*: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe. I believe this might help: Elastic sizing was introduced in Ontap 9.6. This article details the issue: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/Elastic_Sizing_with_FlexGroup_Volumes <https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/Elastic_Sizing_with_FlexGroup_Volumes> This should be turned on by default but can be turned off via: node run nodename flexgroup set elastic-sizing=false On Wed, May 26, 2021 at 12:01 PM Rue, Randy <randyrue@gmail.com <mailto:randyrue@gmail.com>> wrote: OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell? On 5/26/2021 8:47 AM, Rue, Randy wrote: > sure enough, that constituent volume is completely full: > > scarab::> vol show -vserver scharpdata -is-constituent true > Vserver Volume Aggregate State Type Size > Available Used% > --------- ------------ ------------ ---------- ---- ---------- > ---------- ----- > scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB > 0B 100% > > > now what? > > > On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote: >> vol show -vserver vserver_name -is-constituent true _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> https://www.teaparty.net/mailman/listinfo/toasters <https://www.teaparty.net/mailman/listinfo/toasters>
It’s on by default and there’s no straightforward way to disable it, so it’s likely on.
You can verify what it’s set to with:
::> set diag; node run * flexgroup set
That gives a list of current per-node settings. proactive-resizing is what you should see and it should be “true.”
I took a look at the ASUP from scarab. There’s definitely something wonky going on with that FG. No elastic sizing and no proactive sizing happening (note the green line is flat). And member 2 is way more full than the others.
[cid:image001.png@01D75242.7F4B4F00]
So there may be a larger file in that one.
Have a look at this blog to see how to identify files in member volumes:
https://whyistheinternetbroken.wordpress.com/2021/04/27/how-to-map-file-and-...
I’d suggest also opening up a support case.
From: Toasters toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net On Behalf Of Rue, Randy Sent: Wednesday, May 26, 2021 3:03 PM To: Toasters toasters@teaparty.net Subject: Re: WAFL error on a flexgroup?
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Thank you.
I'm not finding any mention of turning this feature on or off, is the default? If so, we're at 9.8P1 and should be running it.
On 5/26/2021 11:44 AM, Parisi, Justin wrote: TR-4571
https://www.netapp.com/us/media/tr-4571.pdf
From: Rue, Randy randyrue@gmail.commailto:randyrue@gmail.com Sent: Wednesday, May 26, 2021 2:43 PM To: Parisi, Justin Justin.Parisi@netapp.commailto:Justin.Parisi@netapp.com; Douglas Siggins siggins@gmail.commailto:siggins@gmail.com Cc: Toasters toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: WAFL error on a flexgroup?
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
OK, after reading the below it sounds like elastic sizing should already be on and should have prevented my full constituent volume. Why didn't it?
What's proactive resizing? Where can I find out more about it?
On 5/26/2021 9:23 AM, Parisi, Justin wrote: I’d actually recommend 9.8 and proactive resizing instead.
Elastic sizing is reactive and incurs a latency penalty when used. And it can’t be used with vol autosize.
Proactive resizing starts resizing when we get to 60% used and maintains a relatively even free space buffer and can be used with autosize.
From: Toasters toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net On Behalf Of Douglas Siggins Sent: Wednesday, May 26, 2021 12:15 PM To: Rue, Randy randyrue@gmail.commailto:randyrue@gmail.com Cc: Toasters toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: WAFL error on a flexgroup?
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I believe this might help:
Elastic sizing was introduced in Ontap 9.6. This article details the issue: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP...
This should be turned on by default but can be turned off via: node run nodename flexgroup set elastic-sizing=false
On Wed, May 26, 2021 at 12:01 PM Rue, Randy <randyrue@gmail.commailto:randyrue@gmail.com> wrote: OK, resizing the flexgroup added some space to the constituent volume (to all of them, actually) and now it has free space. But a look at the eight CVs in the FG shows they range from 22% to 68% full. Is something keeping them more or less balanced? If so, how did it fail? And if not, what the hell?
On 5/26/2021 8:47 AM, Rue, Randy wrote:
sure enough, that constituent volume is completely full:
scarab::> vol show -vserver scharpdata -is-constituent true Vserver Volume Aggregate State Type Size Available Used%
scharpdata scharp_systems__0002 scarab_b_SSD_1 online RW 448GB 0B 100%
now what?
On 5/26/2021 8:38 AM, Vanderyt, Jeff wrote:
vol show -vserver vserver_name -is-constituent true
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
-
Douglas Siggins -
Parisi, Justin -
Rue, Randy -
Vanderyt, Jeff