Indeed. On a 7.3.7P1 immediately after creating 1TB volume with space guarantee “none” this volume consumes approximately 6GB of allocated space:
Volume Allocated Used Guarantee
t 5965988KB 772KB none
which more or less matches inodefile size, although the space actually consumed by it is zero:
16K ---------- 1 root root 6.1G Dec 11 10:11 inofile
I was under impression that inofile grows as needed but probably it changed at some point.
Thank you!
---
With best regards
Andrei Borzenkov
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS
FUJITSU
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com
Web: ru.fujitsu.com
Company details: ts.fujitsu.com/imprint
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Sebastian Goetze
Sent: Thursday, December 10, 2015 7:14 PM
To: Mike Thompson; John Stoffel
Cc: toasters@teaparty.net Lists
Subject: Re: cdot missing disk space
I didn't do the math, but just an idea:
ONTAP calculates an average file size of 32K to determine the number of inodes per volume. Every inode takes 192 Bytes from the INODEFILE. So, if you take 2TB / 32KB * 192B = 12GB... (actually I *did* do the math right now, I was curious...)
It seems this is enforced on *newer* aggregates...
According to KB https://kb.netapp.com/support/index?page=content&id=3011432 you can't go below the VolSize/32K limit.
How about provisioning the volumes small (and thin) and letting them *AutoGrow* to 2TB?
That way, they're not using up space (not even for inodes), yet are able to contain up to 2TB...
Just my 2c
Sebastian
On 12/10/2015 8:43 AM, Mike Thompson wrote:
sample vol and aggr details below
can't upgrade, not on support, we have about 1PB in production across two clusters. I am seeing this effect on both clusters, but not on all filers, which is strange.
if I move vols between affected and unaffected filers/aggrs, the allocated vs used normalizes for the volumes on the unaffected node, then re-inflate when moved back to the original node/aggr
Volume Allocated Used
vol created on problem aggr v164402 11932348KB 1924KB
after move to unaffected aggr v164402 2872KB 2872KB
after move back to orig aggr v164402 12005552KB 75284KB
resized to 1TB, moved to another aggr, and back to orig aggr
v164402 6003080KB 37972KBit appears to be related to the size of the volume. we thin-provision all volumes by setting them to a large size and set space guarantee to none.
vols sized at 2TB end up being allocated 12GB of space when created. vols sized to 1TB end up with 6GB allocated when created, even though they are completely empty.what's strange is this is happening on some filers/aggregates but not others.
these clusters originated running Ontap GX, and then were upgraded to Ontap8 c-mode some time ago.
the aggregates that seem to be experiencing this allocation 'inflation' seem to be those that were created after the cluster was upgraded to Ontap8.the aggregates that were originally created as GX aggrs, have identically matching 'allocated' and 'used' values in 'aggr show_space'
for example:
recently built aggregate under Ontap8:
Aggregate Allocated Used Avail
Total space 18526684036KB 13662472732KB 1233809368KB <- massive difference!aggregate originated under Ontap GX:
Aggregate Allocated Used Avail
Total space 32035928920KB 32035928920KB 1404570796KB <- identical, which is what i would expectmaybe this allocation scheme changed at an aggregate level under Ontap 8? perhaps it's expected behavior?
things do seem to normalize as the volumes begin to fill up though, so I believe that this space is not truly gone permanently, but it certainly appears to be not available for use, since tons of space is allocated to volumes that have very little data in them, and we have a LOT of volumes in these clusters.
it's definitely making it look like we are missing a substantial percentage of our disk space, when trying to reconcile the sum of data used when tallying up volume size, and comparing it to the aggregate used/remaining sizes.example volume and affected aggregate details:
bc-gx-4b::*> vol show v164346 -instance
(volume show)
Virtual Server Name: bc
Volume Name: v164346
Aggregate Name: gx4b_1
Volume Size: 2TB
Name Ordinal: base
Volume Data Set ID: 4041125
Volume Master Data Set ID: 2151509138
Volume State: online
Volume Type: RW
Volume Style: flex
Volume Ownership: cluster
Export Policy: default
User ID: jobsys
Group ID: cgi
Security Style: unix
Unix Permissions: ---rwxr-x--x
Junction Path: /bc/shows/ID2/DFO/0820
Junction Path Source: RW_volume
Junction Active: true
Parent Volume: v164232
Virtual Server Root Volume: false
Comment:
Available Size: 1.15TB
Total Size: 2TB
Used Size: 146.7MB
Used Percentage: 42%
Autosize Enabled (for flexvols only): false
Maximum Autosize (for flexvols only): 2.40TB
Autosize Increment (for flexvols only): 102.4GB
Total Files (for user-visible data): 31876689
Files Used (for user-visible data): 132
Maximum Directory Size: 100MB
Space Guarantee Style: none
Space Guarantee In Effect: true
Minimum Read Ahead: false
Access Time Update Enabled: true
Snapshot Directory Access Enabled: true
Percent of Space Reserved for Snapshots: 0%
Used Percent of Snapshot Reserve: 0%
Snapshot Policy: daily
Creation Time: Tue Dec 08 11:22:58 2015
Language: C
Striped Data Volume Count: -
Striped Data Volume Stripe Width: 0.00B
Current Striping Epoch: -
One data-volume per member aggregate: -
Concurrency Level: -
Optimization Policy: -
Clone Volume: false
Anti-Virus On-Access Policy: default
UUID of the volume: 17fa4c6d-9de1-11e5-a888-123478563412
Striped Volume Format: -
Load Sharing Source Volume: -
Move Target Volume: false
Maximum Write Alloc Blocks: 0
Inconsistency in the file system: false
bc-gx-4b::*> aggr show -aggregate
far4a_1 gx1a_1 gx1a_2 gx1b_1 gx2a_1 gx2b_1 gx3a_1 gx4b_1
near1b_1 near3b_1 root_1a root_1b root_2a root_2b root_3a root_3b
root_4a root_4b slow2a_1 systems
bc-gx-4b::*> aggr show -aggregate gx4b_1
Aggregate: gx4b_1
UUID: c624f85e-96d3-11e3-a6ce-00a0980bb25a
Size: 18.40TB
Used Size: 17.25TB
Used Percentage: 94%
Available Size: 1.15TB
State: online
Nodes: bc-gx-4b
Number Of Disks: 63
Disks: bc-gx-4b:0a.64, bc-gx-4b:0e.80,
...
bc-gx-4b:0a.45
Number Of Volumes: 411
Plexes: /gx4b_1/plex0(online)
RAID Groups: /gx4b_1/plex0/rg0,
/gx4b_1/plex0/rg1,
/gx4b_1/plex0/rg2
Raid Type: raid_dp
Max RAID Size: 21
RAID Status: raid_dp
Checksum Enabled: true
Checksum Status: active
Checksum Style: block
Inconsistent: false
Ignore Inconsistent: off
Block Checksum Protection: on
Zoned Checksum Protection: -
Automatic Snapshot Deletion: on
Enable Thorough Scrub: off
Volume Style: flex
Volume Types: flex
Has Mroot Volume: false
Has Partner Node Mroot Volume: false
Is root: false
Wafliron Status: -
Percent Blocks Scanned: -
Last Start Error Number: -
Last Start Error Info: -
Aggregate Type: aggr
Number of Quiesced Volumes: -
Number of Volumes not Online: -
Number of LS Mirror Destination Volumes: -
Number of DP Mirror Destination Volumes: -
Number of Move Mirror Destination Volumes: -
Number of DP qtree Mirror Destination Volumes: -
HA Policy: sfo
Block Type: 64-bit
On Wed, Dec 9, 2015 at 12:50 PM, John Stoffel <john@stoffel.org> wrote:
Can you post the details of one of these volumes? And of the
aggregate you have them in? It smells like there's some sort of
minimum volume size setting somewhere.
Or maybe there's an aggregate level snapshot sitting around?
Can you upgrade? You're in cluster mode, so hopefully it shoudln't be
too hard to move to 8.1, then 8.2 and onto 8.3, since there's lots of
nice bug fixes.
_______________________________________________Toasters mailing listToasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters
