Apologies if this seems a bit vague - I'm in need of a quick sanity check. Doing a qtree delete - it's taking 'a while'. This seems to be because it's deleting every file at a rate of 50-100 files per second, and there's about a million files in the qtree.
Is that a 'reasonable' rate of deletion? (This is using an API based 'file-delete-file' operation, if that's relevant).
Bonus question would be - does that also apply if you do it via a "qtree delete -f" or is the API delete run at about the same speed?
On 25 September 2015 at 13:31, basilberntsen@gmail.com wrote:
Yes, that seems possible. Operations working on large numbers of small files always take a long time.
Sent from my BlackBerry 10 smartphone on the Bell network. *From: *Edward Rolison *Sent: *Friday, September 25, 2015 8:00 AM *To: *toasters@teaparty.net *Subject: *File delete rate
Apologies if this seems a bit vague - I'm in need of a quick sanity check. Doing a qtree delete - it's taking 'a while'. This seems to be because it's deleting every file at a rate of 50-100 files per second, and there's about a million files in the qtree.
Is that a 'reasonable' rate of deletion? (This is using an API based 'file-delete-file' operation, if that's relevant).
I wonder if (like in Unix), as the number of files left drops, the rate of deletion will increase or go faster. I found in one case that the time was being taken in reading then re-writing the filesystem directory index of files. As the number dropped from 100,000+ files the rate of deletes increased.
On Fri, Sep 25, 2015 at 7:19 AM, basilberntsen@gmail.com wrote:
Pretty sure the only shortcut is volume operations like qtree snapmirror.
Sent from my BlackBerry 10 smartphone on the Bell network. *From: *Edward Rolison *Sent: *Friday, September 25, 2015 8:48 AM *To: *Basil B *Cc: *toasters@teaparty.net *Subject: *Re: File delete rate
Bonus question would be - does that also apply if you do it via a "qtree delete -f" or is the API delete run at about the same speed?
On 25 September 2015 at 13:31, basilberntsen@gmail.com wrote:
Yes, that seems possible. Operations working on large numbers of small files always take a long time.
Sent from my BlackBerry 10 smartphone on the Bell network. *From: *Edward Rolison *Sent: *Friday, September 25, 2015 8:00 AM *To: *toasters@teaparty.net *Subject: *File delete rate
Apologies if this seems a bit vague - I'm in need of a quick sanity check. Doing a qtree delete - it's taking 'a while'. This seems to be because it's deleting every file at a rate of 50-100 files per second, and there's about a million files in the qtree.
Is that a 'reasonable' rate of deletion? (This is using an API based 'file-delete-file' operation, if that's relevant).
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
It's probably a single thread process to do the deletions. If possible, from a client, start multiple deletions deep in the tree. I have had to do this trick with deletions and with rsync in the past Forever the multithreaded-ness
Sent from Mobile Outlook
On Sat, Sep 26, 2015 at 9:16 AM -0700, "Patrick Giagnocavo" xemacs5@gmail.com wrote:
I wonder if (like in Unix), as the number of files left drops, the rate of deletion will increase or go faster. I found in one case that the time was being taken in reading then re-writing the filesystem directory index of files. As the number dropped from 100,000+ files the rate of deletes increased.
On Fri, Sep 25, 2015 at 7:19 AM, basilberntsen@gmail.com wrote: Pretty sure the only shortcut is volume operations like qtree snapmirror. Sent from my BlackBerry 10 smartphone on the Bell network. From: Edward RolisonSent: Friday, September 25, 2015 8:48 AMTo: Basil BCc: toasters@teaparty.netSubject: Re: File delete rate Bonus question would be - does that also apply if you do it via a "qtree delete -f" or is the API delete run at about the same speed? On 25 September 2015 at 13:31, basilberntsen@gmail.com wrote: Yes, that seems possible. Operations working on large numbers of small files always take a long time. Sent from my BlackBerry 10 smartphone on the Bell network. From: Edward RolisonSent: Friday, September 25, 2015 8:00 AMTo: toasters@teaparty.netSubject: File delete rate Apologies if this seems a bit vague - I'm in need of a quick sanity check. Doing a qtree delete - it's taking 'a while'. This seems to be because it's deleting every file at a rate of 50-100 files per second, and there's about a million files in the qtree. Is that a 'reasonable' rate of deletion? (This is using an API based 'file-delete-file' operation, if that's relevant).
_______________________________________________
Toasters mailing list
Toasters@teaparty.net
-
basilberntsen＠gmail.com -
Edward Rolison -
Patrick Giagnocavo -
Tim McCarthy