Looks like we filled up an aggregate that we dump backups into.
Filer bounced itself, to take the aggregate offline, which is an interesting way to deal with the situation:
Aug 6 01:39:46 filer1b [filer1b:wafl.vol.walloc.rsv.offline:error]: Volume backups free space reserve is critically low. Rebooting system to take volume offline. Please contact support for assistance.
Anyway, the aggr is in restricted state now, and when I try to online it, event log says:
8/7/2018 08:43:46 filer1b WARNING wafl.vol.walloc.rsv.failmount: Failing mount of volume backups. Free space reserve is too low. Please contact support for assistance.
I'd really like to not have to blow this aggr away and re-run all these backups. Presumably there's a way to get this back online.
This is on cdot 8.2.4p4
Any ideas?
Contacting support is your best bet here.
From: <toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net> on behalf of Mike Thompson <mike.thompson@gmail.commailto:mike.thompson@gmail.com> Date: Tuesday, August 7, 2018 at 12:21 PM To: "Toasters@teaparty.netmailto:Toasters@teaparty.net" <Toasters@teaparty.netmailto:Toasters@teaparty.net> Subject: wafl.vol.walloc.rsv.failmount
Looks like we filled up an aggregate that we dump backups into.
Filer bounced itself, to take the aggregate offline, which is an interesting way to deal with the situation:
Aug 6 01:39:46 filer1b [filer1b:wafl.vol.walloc.rsv.offline:error]: Volume backups free space reserve is critically low. Rebooting system to take volume offline. Please contact support for assistance.
Anyway, the aggr is in restricted state now, and when I try to online it, event log says:
8/7/2018 08:43:46 filer1b WARNING wafl.vol.walloc.rsv.failmount: Failing mount of volume backups. Free space reserve is too low. Please contact support for assistance.
I'd really like to not have to blow this aggr away and re-run all these backups. Presumably there's a way to get this back online.
This is on cdot 8.2.4p4
Any ideas?
yeah unfortunately this particular cluster not under support, just using it as disk staging for backups.
the data is disposable, so i guess a follow up question might be, how do i go about removing the volumes that are contained on this aggregate from the cluster, while it's in offline or restricted state, and then blow away the aggregate, so i can re-roll it from scratch.
On Tue, Aug 7, 2018 at 12:24 PM, Parisi, Justin Justin.Parisi@netapp.com wrote:
Contacting support is your best bet here.
From: toasters-bounces@teaparty.net on behalf of Mike Thompson < mike.thompson@gmail.com> Date: Tuesday, August 7, 2018 at 12:21 PM To: "Toasters@teaparty.net" Toasters@teaparty.net Subject: wafl.vol.walloc.rsv.failmount
Looks like we filled up an aggregate that we dump backups into.
Filer bounced itself, to take the aggregate offline, which is an interesting way to deal with the situation:
Aug 6 01:39:46 filer1b [filer1b:wafl.vol.walloc.rsv.offline:error]: Volume backups free space reserve is critically low. Rebooting system to take volume offline. Please contact support for assistance.
Anyway, the aggr is in restricted state now, and when I try to online it, event log says:
8/7/2018 08:43:46 filer1b WARNING wafl.vol.walloc.rsv.failmount: Failing mount of volume backups. Free space reserve is too low. Please contact support for assistance.
I'd really like to not have to blow this aggr away and re-run all these backups. Presumably there's a way to get this back online.
This is on cdot 8.2.4p4
Any ideas?
Any other volumes on that aggregate that can move elsewhere? Any other nodes in the cluster with free space?
On Aug 7, 2018, at 11:21 AM, Mike Thompson mike.thompson@gmail.com wrote:
wafl.vol.walloc.rsv.failmoun
Delete all snapshots?
Francis Kim Cell: 415-606-2525 Direct: 510-644-1599 x334 fkim@berkcom.commailto:fkim@berkcom.com www.berkcom.com
On Aug 7, 2018, at 7:29 AM, Mike Gossett <cmgossett@gmail.commailto:cmgossett@gmail.com> wrote:
Any other volumes on that aggregate that can move elsewhere? Any other nodes in the cluster with free space?
On Aug 7, 2018, at 11:21 AM, Mike Thompson <mike.thompson@gmail.commailto:mike.thompson@gmail.com> wrote:
wafl.vol.walloc.rsv.failmoun
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Hey Mike,
some interesting ideas out of this old post:
https://community.netapp.com/t5/Simulator-Discussions/aggr0-full-on-7-3-4-si...
* Try to shrink the root volume * Try to convert the aggregate to raid-4
Worth a try.
Best,
Alexander Griesser Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.commailto:AGriesser@anexia-it.com Web: http://www.anexia-it.comhttp://www.anexia-it.com/
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: toasters-bounces@teaparty.net toasters-bounces@teaparty.net Im Auftrag von Mike Thompson Gesendet: Dienstag, 7. August 2018 18:21 An: toasters@teaparty.net Lists toasters@teaparty.net Betreff: wafl.vol.walloc.rsv.failmount
Looks like we filled up an aggregate that we dump backups into.
Filer bounced itself, to take the aggregate offline, which is an interesting way to deal with the situation:
Aug 6 01:39:46 filer1b [filer1b:wafl.vol.walloc.rsv.offline:error]: Volume backups free space reserve is critically low. Rebooting system to take volume offline. Please contact support for assistance.
Anyway, the aggr is in restricted state now, and when I try to online it, event log says:
8/7/2018 08:43:46 filer1b WARNING wafl.vol.walloc.rsv.failmount: Failing mount of volume backups. Free space reserve is too low. Please contact support for assistance.
I'd really like to not have to blow this aggr away and re-run all these backups. Presumably there's a way to get this back online.
This is on cdot 8.2.4p4
Any ideas?
Thanks for the replies on this one all.
The solution was to run wafliron on the aggregate, which very quickly brought it and it's volumes back online for us.
set diag storage aggregate wafliron start -aggregate backups
Thanks again
On Tue, Aug 7, 2018 at 12:21 PM, Mike Thompson mike.thompson@gmail.com wrote:
Looks like we filled up an aggregate that we dump backups into.
Filer bounced itself, to take the aggregate offline, which is an interesting way to deal with the situation:
Aug 6 01:39:46 filer1b [filer1b:wafl.vol.walloc.rsv.offline:error]: Volume backups free space reserve is critically low. Rebooting system to take volume offline. Please contact support for assistance.
Anyway, the aggr is in restricted state now, and when I try to online it, event log says:
8/7/2018 08:43:46 filer1b WARNING wafl.vol.walloc.rsv.failmount: Failing mount of volume backups. Free space reserve is too low. Please contact support for assistance.
I'd really like to not have to blow this aggr away and re-run all these backups. Presumably there's a way to get this back online.
This is on cdot 8.2.4p4
Any ideas?
-
Alexander Griesser -
Francis Kim -
Mike Gossett -
Mike Thompson -
Parisi, Justin