Hello,
Are there any utilities/scripts out there that will keep track of storage trends on NetApp volumes? I'm in need of something that will show how much volumes have increased in space utilization over time. My first guess was just simply do 'df' RSH commands and parse/store the data, but I'd like to know if there was anything more robust before I began.
TIA,
Daniel
______________________________________________________________________ This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
You can use Operations Manager (formerly Data Fabric Manager - DFM) from NetApp that will do all of the trending reports that you need. Sent from my Verizon Wireless BlackBerry
-----Original Message----- From: "Daniel Keisling" daniel.keisling@austin.ppdi.com
Date: Tue, 11 Dec 2007 11:45:30 To:toasters@mathworks.com Subject: Tracking Space Trends on NetApp Filers
Hello, Are there any utilities/scripts out there that will keep track of storage trends on NetApp volumes? I'm in need of something that will show how much volumes have increased in space utilization over time. My first guess was just simply do 'df' RSH commands and parse/store the data, but I'd like to know if there was anything more robust before I began. TIA,
Daniel
______________________________________________________________________ This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
If you just want to monitor change over time and are not looking for predictive trend analysis, Toasterview might work for you:
http://www.cuddletech.com/code/toasterview/index.html
That's a freebie.
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Andre.Clark@earthlink.net Sent: Tue 12/11/2007 11:01 AM To: Daniel Keisling; owner-toasters@mathworks.com; toasters@mathworks.com Subject: Re: Tracking Space Trends on NetApp Filers
You can use Operations Manager (formerly Data Fabric Manager - DFM) from NetApp that will do all of the trending reports that you need. Sent from my Verizon Wireless BlackBerry
-----Original Message----- From: "Daniel Keisling" daniel.keisling@austin.ppdi.com
Date: Tue, 11 Dec 2007 11:45:30 To:toasters@mathworks.com Subject: Tracking Space Trends on NetApp Filers
Hello,
Are there any utilities/scripts out there that will keep track of storage trends on NetApp volumes? I'm in need of something that will show how much volumes have increased in space utilization over time. My first guess was just simply do 'df' RSH commands and parse/store the data, but I'd like to know if there was anything more robust before I began.
TIA,
Daniel
______________________________________________________________________ This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
Herring, Jason wrote:
If you just want to monitor change over time and are not looking for predictive trend analysis, Toasterview might work for you:
http://www.cuddletech.com/code/toasterview/index.html
That's a freebie.
ToasterView is nice, but beware that it has issues grokking vols/aggrs
2TB. See http://toasters.mathworks.com/toasters/16940.html for more
information on the specifics of the MIB-issues.
The reply in that thread says that Daniel Leeds was updating the code in 2006 to use the new MIB values for greater than 2TB vols/aggrs. Was that ever completed and posted? If so, someone should forward it to Ben Rockwood (benr@cuddletech.com) who created/maintains Toasterview so the code base is updated as well.... gotta luv open source!
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Nick Silkey Sent: Tue 12/11/2007 1:37 PM To: toasters@mathworks.com Subject: Re: Tracking Space Trends on NetApp Filers
Herring, Jason wrote:
If you just want to monitor change over time and are not looking for predictive trend analysis, Toasterview might work for you:
http://www.cuddletech.com/code/toasterview/index.html
That's a freebie.
ToasterView is nice, but beware that it has issues grokking vols/aggrs
2TB. See http://toasters.mathworks.com/toasters/16940.html for more
information on the specifics of the MIB-issues.
Here's a simple perl script which I wrote to see how my Aggregates and Volumes on my R200 were balancing out, since I needed to grow a volume, but the plain output of 'df' is useless to tell you which Aggregate actually holds the volume. So you need to use the 'vol status' command to figure this out and some funky parsing, etc.
This script makes my life easier, but it does zero trend analysis or anything like that, it's just an easy to use df-like tool that fits my needs.
#!/usr/local/bin/perl -w # # Version 1.0 - December 12th, 2007 # License: GPLv2
use strict; #no warnings 'closure'; use Getopt::Long; &Getopt::Long::Configure("no_ignore_case");
my $DEBUG = -1;
my $rshcmd = "/bin/rsh"; my $rshargs = ""; my $user = "root";
my $ret = GetOptions( 'D:i' => $DEBUG, 'e=s' => $rshcmd, 'u=s' => $user, );
# Default DEBUG level is 1 if ($DEBUG == 0) { $DEBUG++; }
&usage if (!$ret); &usage if ($#ARGV < 0);
if ($rshcmd =~ m/ssh/) { $rshargs = "-l $user"; }
if (! -e $rshcmd) { print STDERR "Error! Cannot execute $rshcmd, please check your path.\n\n"; &usage; exit; }
foreach my $filer (@ARGV) { my %data = (); my %aggr = ();
my $getaggrdf = 0; my $dfargs = "";
my @rawstatus = &getvolstatus($filer,$rshargs); my @rawdf = &getvoldf($filer,$rshargs,$dfargs); $dfargs = "-A"; my @aggrdf = &getvoldf($filer,$rshargs,$dfargs); %data = &parsestatus(%data,$filer,@rawstatus); %data = &parsedf(%data,$filer,$getaggrdf,@rawdf); $getaggrdf++; %aggr = &parsedf(%data,$filer,$getaggrdf,@aggrdf); &showdata($filer,%data,%aggr); undef %aggr; undef %data; }
exit;
#--------------------------------------------------------------------- sub DBG { my $l = 0;
if ($#_ == 1) { $l = shift @_; } # Indent Debug output for each level we go up. print STDERR " "x$DEBUG, @_ if ($DEBUG >= $l); }
#--------------------------------------------------------------------- sub getvolstatus { my $filer = shift @_; my $rshargs = shift @_;
DBG 1, "getvolstatus($filer,$rshargs)\n";
open(IN,"$rshcmd $rshargs $filer vol status -v|") || die "Error! Cannot $rshcmd $rshargs $filer: $!\n"; my @d = <IN>; close(IN); DBG 5, "Found $#d lines of input.\n"; return @d; }
#--------------------------------------------------------------------- sub getvoldf { my $filer = shift @_; my $rshargs = shift @_; my $dfargs = shift @_;
DBG 1, "getvoldf($filer,$rshargs,$dfargs)\n";
open(IN,"$rshcmd $rshargs $filer df $dfargs|") || die "Error! Cannot $rshcmd $filer: $!\n"; my @d = <IN>; close(IN); DBG 5, "Found $#d lines of input.\n"; return @d; }
#--------------------------------------------------------------------- sub parsestatus {
my $ref = shift @_; my $filer = shift @_;
DBG 1, "parse($filer)\n";
my %d = %$ref; my $vol;
STLOOP: foreach (@_) { chomp; DBG 4, "ST: $_\n";
next STLOOP if (m/Volume State Status\s+Options$/);
if (m/^\s*([\w_]+) online (.*)$/) { $vol = $1; DBG 5, "Volume: $vol\n"; my $opt = $2; $d{$vol}->{AGGR} = "";
# Parse out any remaining options on the line... if ($opt ne "") { DBG 6, "opt: $opt\n"; my @o = split(",",$opt); PAIR: foreach my $pair (@o) { if ($pair =~ m/\w+=\w+/) { my ($opt, $val) = split("=",$pair); DBG 6, "option: $opt = $val.\n"; $d{$vol}{$opt} = $val || ""; } } } next STLOOP; }
if (m/Containing aggregate: (.*)/) { my $aggr = $1; $aggr =~ s/'|\s//g;
$d{$vol}{AGGR} = $aggr; next STLOOP; }
if (m/,$/) { my @o = split(",",$_); foreach my $pair (@o) { my ($opt, $val) = split("=",$pair); $d{$vol}{$opt} = $val; DBG 5, "option: $opt = $val.\n"; } } } return %d; }
#--------------------------------------------------------------------- sub parsedf {
my $ref = shift @_; my $filer = shift @_; my $doaggr = shift @_;
DBG 1, "parsedf($filer,$doaggr)\n";
my %d = %$ref; my $vol; my $aggr;
DFLOOP: foreach (@_) { chomp;
if ($doaggr) { DBG 3, "DFA: $_\n"; next DFLOOP if (m/^Aggregate\s+kbytes\s+/); next DFLOOP if (m@/.snapshot\s+@);
if (m/^(\w+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+%)/) { DBG 3, "Aggr = $1\n"; $d{$1}{ATOTAL} = $2; $d{$1}{AUSED} = $3; $d{$1}{AAVAIL} = $4; $d{$1}{ACAP} = $5; next DFLOOP; } } else { DBG 3, "DF: $_\n"; next DFLOOP if (m/^Filesystem\s+total\s+used\s+avail\s+capacity\s+Mounted on$/); next DFLOOP if (m/^snap reserve/);
if (m@^/vol/(.*)/\s+(\d+)\s+(\d+)\s+(\d+)\s+(.+%)@) { $vol = $1; my $total = $2; my $used = $3; my $avail = $4; my $cap = $5; DBG 2, "Volume: $vol, T: $total, U: $used, A: $avail, C: $cap\n"; $d{$vol}{DFTOTAL} = $total; $d{$vol}{DFUSED} = $used; $d{$vol}{DFAVAIL} = $avail; $d{$vol}{DFCAP} = $cap; next DFLOOP; } } } return %d; }
#--------------------------------------------------------------------- sub showdata {
DBG 2, "showdata()\n"; my $filer = shift @_; my $href = shift @_; my %d = %$href;
my $aref = shift @_; my %a = %$aref;
my %shown;
print "\nFiler: $filer\n\n";
printf "%-8s %-30s %8s %8s %4s\n", "Aggr","Volume", "Used", "Total", "Pcnt"; printf "%-8s %-30s %8s %8s %4s\n", "-"x8,"-"x25,"-"x8,"-"x8,"-"x4;
my $blank = ""; foreach my $vol (sort { $d{$a}{AGGR} cmp $d{$b}{AGGR} || $d{$b}{DFTOTAL} <=> $d{$a}{DFTOTAL} } keys %d) {
# All this to print a seperator between aggregates... if ($d{$vol}{AGGR} ne "$blank" ) { my $aggr = $d{$vol}{AGGR}; if ($a{$aggr}) { printf "\n%-8s %-30s %8s %8s %4s\n", $aggr, "--------", &scale($a{$aggr}{AUSED}), &scale($a{$aggr}{ATOTAL}),$a{$aggr}{ACAP}; } $blank = $d{$vol}{AGGR}; }
# Show the data printf "%-8s %-30s %8s %8s %4s\n", $d{$vol}{AGGR}, $vol, &scale($d{$vol}{DFUSED}), &scale($d{$vol}{DFTOTAL}), $d{$vol}{DFCAP}; } }
#--------------------------------------------------------------------- # Takes Kilobyte input and scales to MB, GB or higher.
sub scale { my $n = shift @_;
my @suffix = qw (KB MB GB TB); my $i = 0;
while($n > 10000) { $n /= (1024); $i++; } return sprintf("%5.1f$suffix[$i]",$n); }
#--------------------------------------------------------------------- sub usage {
print STDERR <<EOF;
netapp-vol-to-aggr [options] <filer> [<filer> ...]
Print a report of the usage of aggregates and the volumes on those aggregates to make it easier to see how to balance disk useage amongst FlexVols.
Requires that the script be run on a host which has rsh (default) or ssh access to the filer to run the 'df' and 'vol status' commands.
NOTE! You will be prompted three times per-filer for SSH passwords if used, sorry.
Options: -D [#] Print out debugging info, higher numbers print more. -e rsh|ssh Use a specific rsh or ssh transport. -u user For ssh specify the username to use, default is 'root'
Author: John Stoffel (john.stoffel@taec.toshiba.com || john@stoffel.org)
EOF }
DFM\Operations Manger does this.
________________________________
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Daniel Keisling Sent: Tuesday, December 11, 2007 12:46 PM To: toasters@mathworks.com Subject: Tracking Space Trends on NetApp Filers
Hello,
Are there any utilities/scripts out there that will keep track of storage trends on NetApp volumes? I'm in need of something that will show how much volumes have increased in space utilization over time. My first guess was just simply do 'df' RSH commands and parse/store the data, but I'd like to know if there was anything more robust before I began.
TIA,
Daniel
______________________________________________________________________ This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
Depending on your required granularity, Ops Manager (aka DFM) can show you usage over time at the Qtree level. C-
Daniel Keisling wrote:
Hello,
Are there any utilities/scripts out there that will keep track of storage trends on NetApp volumes? I'm in need of something that will show how much volumes have increased in space utilization over time. My first guess was just simply do 'df' RSH commands and parse/store the data, but I'd like to know if there was anything more robust before I began.
TIA,
Daniel
This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
We Installed "MRTG for Filers" from the NOW site (http://www.netapp.com/news/techontap/0506tot_monthlytool_mrtg.html) Absolutely simple to setup and run, just the way I like it. Not a feature rich as Cricket, Cacti or Nagios, but you can't beat it for pure historical disk usage stats.
Brett
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Daniel Keisling Sent: Wednesday, 12 December 2007 4:46 AM To: toasters@mathworks.com Subject: Tracking Space Trends on NetApp Filers
Hello,
Are there any utilities/scripts out there that will keep track of storage trends on NetApp volumes? I'm in need of something that will show how much volumes have increased in space utilization over time. My first guess was just simply do 'df' RSH commands and parse/store the data, but I'd like to know if there was anything more robust before I began.
TIA,
Daniel
______________________________________________________________________ This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
________________________________ This e-mail and its attachments are intended for the named addressee only and no liability is accepted for use or reliance on any part of this e-mail by any other person. It is confidential, may be subject to privilege and is also subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. Any confidentiality or privilege is not waived or lost because this e-mail has been received by you and you are not the intended recipient. If you are not the intended recipient, please let us know by reply e-mail.
Please note that e-mails can be interfered with, can contain computer viruses or other defects and may not be successfully replicated on other systems. This footnote confirms that this e-mail message has been swept for the presence of computer viruses. However whilst the sender has taken reasonable precautions to minimise the risk of this email and any attachment containing viruses, we cannot accept liability for any such viruses and we give no warranties in relation to any of the above matters. If you have any doubts about the authenticity of this e-mail please contact the sender immediately. No responsibility is accepted for any changes made to a document other than those made by the sender.
depending on your backup software you can probably gather the same information from logs of full backups over time.
Daniel Keisling wrote:
Hello,
Are there any utilities/scripts out there that will keep track of storage trends on NetApp volumes? I'm in need of something that will show how much volumes have increased in space utilization over time. My first guess was just simply do 'df' RSH commands and parse/store the data, but I'd like to know if there was anything more robust before I began.
TIA,
Daniel
This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
From what I can find, if a filer is not part of an AD forest, it will not listen on port 445 for SMB requests. Is there a way to change so that it WILL listen on port 445?
I know there is a way to turn it off, but not to force it. We have a process that attempts to contact the windows servers on 445 first and then falls back to 139 if it doesn't work on 445. It causes a small delay, but we have thousands of attempts an hour and even a small delay adds up. We have currently hacked a perl module to get around the issue, but I was hoping there was something we could do on the netapp side.
Thanks Jack
445 is used for KRB requests. You (to my knowledge) cannot get CIFS + KRB in a Windows environment without being in an AD forest. I don't believe there is any way to force the filer to use 445 without being in an AD forest.
(I'd assume you can get Linux to run SAMBA with KRB outside of an AD forest, but that's a completely different story.)
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Jack Lyons Sent: Wednesday, December 12, 2007 8:50 PM To: toasters@mathworks.com Subject: making filer listen for smb requests on TCP/445
From what I can find, if a filer is not part of an AD forest, it will not listen on port 445 for SMB requests. Is there a way to change so that it WILL listen on port 445?
I know there is a way to turn it off, but not to force it. We have a process that attempts to contact the windows servers on 445 first and then falls back to 139 if it doesn't work on 445. It causes a small delay, but we have thousands of attempts an hour and even a small delay adds up. We have currently hacked a perl module to get around the issue, but I was hoping there was something we could do on the netapp side.
Thanks Jack
Not sure if this will help, but I found this on the NetApp site;
How to force SMB over TCP when NetBIOS is disabled on an Active Directory domain controller
Symptoms
How to force SMB over TCP when NetBIOS is disabled on an Active Directory domain controller
Domain controllers in a Windows Active Directory domain may have NetBIOS disabled. The filer will still attempt to use NetBIOS to communicate with the domain controllers and may time out when port 139 is blocked.
How can you force the filer to use Kerberos and port 445 to communicate with the domain controllers instead of NetBIOS and port 139?
Cause of this problem
Background Information: SMB over TCP vs. SMB over NBT
The SMB (Server Message Block) protocol is used among other things for file sharing in Windows NT / 2000. In Windows NT it ran on top of NBT (NetBIOS over TCP/IP), which used ports 137, 138 (UDP) and 139 (TCP). In Windows 2000, Microsoft added the possibility to run SMB directly over TCP/IP, without the extra layer of NBT. For this they use TCP port 445.
If the domain controller has NBT enabled, it listens on UDP ports 137, 138, and on TCP ports 139, 445. If it has NBT disabled, it listens on TCP port 445 only.
Solution
Setting the option cifs.netbios_over_tcp.enable on the filer to off causes the filer to initiate contact with the filer using DNS and 'SMB over TCP' instead of 'SMB over NBT' and broadcast / WINS lookups.
To set the option to off, at the filer console enter:
options cifs.netbios_over_tcp.enable off
Does this mean that the filer still has to be in an AD forest? Don't know...but with the above configurations is seems like the filer will listen only to the 445 port.
James
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Glenn Walker Sent: Wednesday, December 12, 2007 7:24 PM To: Jack Lyons; toasters@mathworks.com Subject: RE: making filer listen for smb requests on TCP/445
445 is used for KRB requests. You (to my knowledge) cannot get CIFS +
KRB in a Windows environment without being in an AD forest. I don't
believe there is any way to force the filer to use 445 without being in
an AD forest.
(I'd assume you can get Linux to run SAMBA with KRB outside of an AD
forest, but that's a completely different story.)
-----Original Message-----
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com]
On Behalf Of Jack Lyons
Sent: Wednesday, December 12, 2007 8:50 PM
To: toasters@mathworks.com
Subject: making filer listen for smb requests on TCP/445
From what I can find, if a filer is not part of an AD forest, it will
not listen on port 445 for SMB requests. Is there a way to change so
that it WILL listen on port 445?
I know there is a way to turn it off, but not to force it. We have a
process that attempts to contact the windows servers on 445 first and
then falls back to 139 if it doesn't work on 445. It causes a small
delay, but we have thousands of attempts an hour and even a small delay
adds up. We have currently hacked a perl module to get around the
issue, but I was hoping there was something we could do on the netapp
side.
Thanks
Jack