toasters

toasters@lists.teaparty.net

13531 discussions

kicking an unhappy ssh daemon
by David N. Blank-Edelman 29 Jun '11

29 Jun '11

Hi- We have an interesting situation with one of our filers. Earlier today, one of our monitoring systems managed to eat up all of the available ssh slots on the box. Once we got that under control, we have found we still can't get on to the box via ssh or ssh commands to it, they just hang. There is something listening and accepting connections, but it hangs once the connection is made. We've tried: - turning off and on the ssh options - changing the ssh port (and changing it back) - secureadmin setup ssh (and moving the keys back) Can you think of anything else I could do to kick the daemon short of rebooting the box (which is currently serving in production)? Perhaps something in a super-squirrel-secret priv mode? Thanks! -- dNb

4 4

Re: kicking an unhappy ssh daemon
by Bennett Todd 29 Jun '11

29 Jun '11

That suggests an idea for the possible cause, to me. Symptom: inbound ssh connections start hanging, and eventually, after a long time, recover. Each new ssh connection requires some high-quality random bits, unguessable absent knowledge that can't be externally observed. Suppose the ssh server is drawing them directly from an entropy pool that is only slowly, gradually refilled by the OS observing things like the rotational latency of spinning drives as that's slightly disturbed by turbulence? I don't know the OS that underlies netapps, so I don't know if that pool can be manually fed from commandline, nor if the ssh server can be redirected to a PRNG itself just seeded by the entropy pool. Or if so how to do it. -Bennett On Jun 29, 2011 1:31 AM, "Stefan Funke" <bundy(a)usage.de> wrote:

1 0

Re: kicking an unhappy ssh daemon
by David N. Blank-Edelman 29 Jun '11

29 Jun '11

Hi Felix- Thanks for your response. On Tuesday, June 28, 2011 at 6:08 PM, Felix Schröder wrote: > To kick out a stalled SSH session just type "ssh root@filer logout telnet" on a *nix machine, then type in the password - That will force the filer to logout any interactive (ssh) session. It works the same way with rsh. I must not have been clear in my message. Both interactive, and non-interactive ssh command the machine are hanging. I can't even do an ssh netapp help, nevermind execute the logout command. The connections never get that far. I should also mention that at the moment, according to netstat, there are no existing ssh connections so there isn't really anything to logout. -- dNb

2 1

Re: kicking an unhappy ssh daemon
by David N. Blank-Edelman 29 Jun '11

29 Jun '11

On Tuesday, June 28, 2011 at 7:48 PM, Felix Schröder wrote: > When a session has become hung, if a serial console session is possible, or > some other means of issuing commands is possible, then the hung connection > may be located in the output of the "netstat -a" command so that the "Swind" > size may be checked: if it is zero, then this problem may be occurring, and > the user may try to close and then re-open the session from the client. Hmm, so my read of this bug is "sometimes sessions in progress hang. If that happens, start another session" which doesn't seem quite like what I am seeing where I can't even start a session at this point. -- dNb

1 0

Issues with Compression?
by Milazzo Giacomo 25 Jun '11

25 Jun '11

Hi all, I've received a couple of alert from field with users using DoT 8.0.1 compression feature on Oracle production environment. Their application hang! NetApp representative here in Italy suggested us to avoid, for the moment, to use compression on Oracle and generally on DB/backup environment... Do you have experience? Positive and not I mean. And, of course, not only on Oracle/DB environment. Thank you in advance,

2 1

one cifs share from two vols ?
by Chaim Rieger 22 Jun '11

22 Jun '11

say i have vol/vol1 and vol/vol2 i want to create a cifs share \\cifs\share1 that shares out /vol/vol1 as \\cifs\share1 and /vol/vol2 as \\cifs\share1\share2 is this possible or would i have to create a second share \\cifs\share1\share2 for the second volume ?

2 1

volume, qtree snapmirror resync question
by Steve Losen 20 Jun '11

20 Jun '11

Hi Toasters, We have recently upgraded to ONTAP 8.0.1 and are migrating to 64 bit aggrs/vols. Quick question: Say I have 32 bit volume A on primary storage being volume snapmirrored to 32 bit volume B on secondary storage. A has one qtree, then entire volume itself. I create 64 bit volume C on secondary storage and use qtree snapmirror to copy /vol/B/- to /vol/C/qtree Is there any way to resync C and A now? Will this work on secondary? snapmirror resync -S primary:/vol/A/- /vol/C/qtree I suppose I must snapmirror break the B, C relationship first. If this will work then we can migrate a lot of our secondary storage to 64 bit without heavily impacting primary storage. Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

1 0

Truncated SSH ouptut on 8.x?
by A Darren Dunham 07 Jun '11

07 Jun '11

Anyone seen differences with SSH command output on 8.x filers? I've encountered this a few times lately, and as best I can tell it's always been on 8.x machines. It appears to be more likely to happen on the first command to a filer in a while. I'll run something like "ssh <filer> exportfs" and either get a line or two or just a normal prompt returned with no output. Never an error. Then I'll just repeat the last command and get full output. In fact I can repeat it multiple times and never see the error again. There is a KB article that seems similar (https://kb.netapp.com/support/index?page=content&id=2013198), but it's not really the same, affects 7.x, and the variable that is mentioned to resolve doesn't seem appropriate. I don't have a lot of hard data on this, I'm not prepared to open a ticket on just yet. But I'm wondering if others have seen it. Thanks, -- Darren

1 0

RE: Hot remove disk shelves?
by Steiner, Jeffrey 02 Jun '11

02 Jun '11

I've heard that asked before, and the answer was no. The issue was something about internal device maps. The filers build maps of which devices are where on the loops, and that includes the shelf itself as well as the drives. Both controllers are aware of the shelves at all times, even if they aren't actually using disks inside them. The only safe way to remove the shelf is at a time when neither controller is aware of it, and that means when both are off. >-----Original Message----- >From: Borzenkov, Andrey [mailto:andrey.borzenkov@ts.fujitsu.com] >Sent: Thursday, June 02, 2011 4:06 PM >To: Steiner, Jeffrey; Blake Golliher; Steve Losen >Cc: toasters(a)mathworks.com >Subject: RE: Hot remove disk shelves? > >Would it be possible via takeover/giveback? I.e. - takeover node2, >unplug shelf on node2 side then do the same with node1? Would it still >lead to problem mentioned? > > >--- >With best regards > >Andrey Borzenkov >Senior system engineer >Service operations > >-----Original Message----- >From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] >On Behalf Of Steiner, Jeffrey >Sent: Thursday, June 02, 2011 6:45 PM >To: Blake Golliher; Steve Losen >Cc: toasters(a)mathworks.com >Subject: RE: Hot remove disk shelves? > >It may appear to work at first, but you'll have a panic and crash >eventually. It might last a week or two, but you'll eventually have >problems. > >If you move quick and you have your FC/NFS timeouts set right, you can >shut down, decable the shelves, and turn it back on before hosts will >notice. I've done that myself more than a few times. > > >>-----Original Message----- >>From: Blake Golliher [mailto:thelastman@gmail.com] >>Sent: Thursday, June 02, 2011 3:07 PM >>To: Steve Losen >>Cc: toasters(a)mathworks.com >>Subject: Re: Hot remove disk shelves? >> >>I've seen people do this without problems, but I've not had to do it >>yet myself. Having the disks unassigned is a good idea since you'll >>want them unowned when you add them to other heads. If you can afford >>to do it while in takeover mode that probably couldn't hurt. >> >>Let us know how it goes. I'm under the impression this works, but is >>unsupported from netapp eng still. >> >>-Blake >> >>On Thu, Jun 2, 2011 at 6:03 AM, Steve Losen ><scl(a)sasha.acc.virginia.edu> >>wrote: >>> >>> Hello Toasters, >>> >>> We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX) >>> attached to the end of a chain where none of the disks are in use >>> (all are spares, all have ownership assigned). We are running >>> ONTAP 8.0.1. >>> >>> The chain is singly attached to both controllers, i.e., the >>> shelf "A" modules are chained to a single FC adapter on one >>> controller and the shelf "B" modules are chained to a single >>> FC adapter on the other controller. >>> >>> The shelves we want to remove are at the far end of the chain. >>> Essentially we want to undo the result of a hot shelf add without >>> downtime. >>> >>> Is it possible to hot remove these shelves? Would it help to >>> "un-own" the disks first? >>> >>> I suppose this could be done using CF takeover/giveback >>> but that might be too disruptive. >>> >>> Steve Losen scl(a)virginia.edu phone: 434-924-0640 >>> >>> University of Virginia ITC Unix Support >>> >>> >>> >

2 1

Hot remove disk shelves?
by Steve Losen 02 Jun '11

02 Jun '11

Hello Toasters, We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX) attached to the end of a chain where none of the disks are in use (all are spares, all have ownership assigned). We are running ONTAP 8.0.1. The chain is singly attached to both controllers, i.e., the shelf "A" modules are chained to a single FC adapter on one controller and the shelf "B" modules are chained to a single FC adapter on the other controller. The shelves we want to remove are at the far end of the chain. Essentially we want to undo the result of a hot shelf add without downtime. Is it possible to hot remove these shelves? Would it help to "un-own" the disks first? I suppose this could be done using CF takeover/giveback but that might be too disruptive. Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

5 4

Moving disks to another controller (head)
by Steve Losen 18 May '11

18 May '11

Hi Toasters, I need to physically move some disks/aggregates to another controller. Do you see anything wrong with this procedure? 1) upgrade DOT to the same version on both heads (7.3.3) 2) shut down and power off both heads 3) physically move the disk shelves 4) boot the destination head in maintenance mode and use "disk assign" to take ownership of the moved disks 5) boot the destination head normally 6) aggr online the foreign aggregates I am concerned about step 4 above. Will it work or do I need to boot the source filer in maintenance mode and change the disk ownership to the destination head before moving the disks? In step 5 some of the foreign aggregates will have names that conflict with native aggregates. I presume that when ONTAP boots it will automatically rename the foreign aggrs as necessary. Or must I rename them on the source filer beforehand? I know that some of you do this routinely, so I appreciate any words of advice. Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

4 3

Partner WWNN must match local WWNN to start FCP under Single Image
by Chaim Rieger 13 May '11

13 May '11

getting this error when trying ti start fcp cf is disabled and not licensed any ideas ?

1 0

Re: Snapmirror to Tape - crazy idea
by John Stoffel 11 May '11

11 May '11

Mike> The only thing to be aware of with LREP is that it is Qtree Mike> level replication. If you want volume Snapmirror (so your Mike> snapshots are also mirrored), LREP will not be too helpful. I'm volume snapmirror. And from all I've read and seen, there's no way to do incremental volume snapmirrors to tape (or LREP). Oh well, no real surprise. John

1 0

Snapmirror to Tape - crazy idea
by John Stoffel 10 May '11

10 May '11

Guys, Has anyone ever tried to do a snapmirror to tape of a volume, then you retrieve it onto the destination. Then can you do the next snapmirror update for the volume to tape as well? The idea is that it takes so much time to write and read 10Tb of tape, not even counting the FedEx across country time, that it might make sense to also write the next snapmirror update to tape as well. God knows I'm tempted to try this... John

3 2

Secure Share Access program on W2K8 server issue
by Paul Letta 28 Apr '11

28 Apr '11

Hi, Is there any one out there using Secure Share Access to view/set file properties in a volume with unix style permissions from a Windows client ? We've used this for years with no issue. But installing the latest Secure Share (3.3) onto Win2K8 server machines doesn't work like it used to. The software installs fine, but when I right click on a file and go to properties, I do not see the SecureShare tab. Any suggestions ? Thanks, Paul

2 2

Procedure for moving volumes on nearstore
by Steve Losen 15 Apr '11

15 Apr '11

Hello Toasters, This is just a little procedure that I figured out by trial and error. I needed to migrate some snapmirror destination volumes on a nearstore to other aggregates on the same nearstore. With a few modifications, this procedure will also work when moving a volume to a different nearstore filer. To avoid load on the primary storage filer I ran the full snapmirror copy (snapmirror initialize) locally on the nearstore filer. However, this left me with a snapmirror snapshot that was not deleted automatically and required deletion by hand. Here is the procedure. Assume oldvol on a nearstore filer is being snapmirrored from a primary storage filer. You want to move oldvol to a different aggregate on the nearstore. On the nearstore (destination) filer do this: vol create newvol newaggr XXXXg vol restrict newvol Now do the full snapmirror locally: snapmirror initialize -S oldvol newvol You may want to comment out your snapmirror.conf line for oldvol to avoid numerous logged errors if a scheduled snapmirror (from primary) of oldvol fails due to oldvol being busy. When the full copy is done, replace oldvol with newvol. vol offline oldvol vol destroy oldvol vol rename newvol oldvol Un-comment your snapmirror.conf line for oldvol and wait a bit for the filer to notice the change. Then snapmirror update from primary storage: snapmirror update oldvol Yes, this actually works. You want to update so that another snapmirror snapshot is created in oldvol. In addition to this snapshot you have a previous snapmirror snapshot that is locked and will not be deleted automatically. snap list -n oldvol The locked snapshot is marked "(snapmirror)". You need to delete it by hand, but you cannot do this without breaking the mirror: snapmirror break oldvol snap delete oldvol lockedsnapshotname Also delete this same snapshot from the source volume. Run this on the primary filer: snap delete oldvol lockedsnapshotname Now re-establish the mirror on the nearstore: snapmirror resync oldvol And you're done. Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

1 0

Assign a spare disk on a V3170?
by Robert McDermott 11 Apr '11

11 Apr '11

Hello Toaster Administrators, We run a 3170 in front of another vendor's disks and recently had a panic/failover event where we didn't get any crash dump files because there was no spare disk available to save to. Technically the 3170 has no spare disks because it runs RAID0 on all the "disks" it sees presented from the SAN (a 3Par T800) and redundancy is all managed on the T800. NetApp has told me we need to have some spare disks available. I've provisioned two 20GB "disks" and exported them to the 3170. Seems like all I need to do now is assign them (auto assign is not on), one to each filer node. But am I correct that the next time I carve out real (bigger) space and go to add it to some aggregate, those crashdump disks will also be available to add? This makes me cringe a little as we're in the habit of using the defaults on the "add array LUNs" wizard including any size disk and we confirm that the number of available disks is the same as whatever we just deployed for the addition. To avoid using the dump disks, we'd need to (as one idea) filter the wizard to only use certain size disks. This is really just a training issue but I can still see the day clearly when someone spaces and adds a 20GB disk to an aggregate that then can't be removed. Messy messy messy.... Is there a way to "hard" assign a disk as a spare? So that it won't be available for addition to an aggregate later? Hope to hear from you... Randy Rue Fred Hutchinson Cancer Research Center Seattle, WA USA

2 1

Problems posting to the toasters list
by Robert McDermott 11 Apr '11

11 Apr '11

I have two colleagues that are subscribed to the toasters list but over the past year haven't been able to successfully post to it; while I can. They send a message to the list, receive no bounce message and the their message never makes it to the list. Their email addresses are: rrue(a)fhcrc.org and skendric(a)fhcrc.org. Does anybody know how to solve this problem? Thanks, -Robert

1 0

iSCSI settings? where are they stored or backuped?
by René Bormann 11 Apr '11

11 Apr '11

> A Darren Dunham <mailto:ddunham@taos.com> > Freitag, 8. April 2011 17:48 > > > > Wow. Lots of really good advice from several folks. I have it working > on the command line as well, so that makes some things easier. It seems > that if I have one or two option areas that I care about, having a > specific script to examine might be better than asking DFM to generate a > full comparison report (which will have lots of differences that I might > not care about). > > Got all the plugins. Almost all the data received. Now I just have to > figure out why a few filers that otherwise report fine in Ops manager > don't get the detailed information for the configurations. > > $ dfm config dump <filer> > Error: Failed to pull configuration from <filer>(3972). > Unable to open /vol/vol0/etc/dfm-10.0.36.39-04082011084439.cfg: No > such file or directory > Error: Failed to get configuration dump from <filer>. > > Thanks again for all the pointers > > ------------------------------------------------------------------------------------- > > Schneidewind, Scott <mailto:Scott.Schneidewind@netapp.com> > Donnerstag, 7. April 2011 23:40 > > > Operations Manager (DFM) offers the ability to compare the > configurations of all the systems, and report on any differences. > > > 1. Download DFM plugins for all relevant versions of Data ONTAP. > > 2. In the DFM GUI, go to Management > Storage System > Configuration > Files > > 3. Create a new configuration file and call it something similar to > baseline.options.config. > > 4. In DFM GUI, go to Management > Storage System > Compare > Configurations > > 5. Compare the baseline.options.config to the other filers that should > have the same configuration information. > > 6. Refresh and click on the completed job to view any differences > > > -Scott > > -----Original Message----- > From: A Darren Dunham [mailto:ddunham@taos.com] > Sent: Thursday, April 07, 2011 2:10 PM > To: toasters(a)mathworks.com > Subject: Easy way to monitor options? > > I would like to check the options on my filers for various items, but > I'm not sure the best way to do it. Could anyone tell me if I'm missing > something? > > SNMP and Operations Manager: I don't see that I can pull the data from > either of these sources. > > SSH and SDK access: I could create an account, but it seems to me that > the only > restriction available is the ability to run the "options" command. > There's no way to grant an account read-only access to the data. I'd > rather not give the monitor direct access to modify things. > > NFS: The current options appear to be visible as a local file in /etc, > but creating NFS access to vol0 for lots of filers for a single file > seems more of a hassle to me. > > At the moment I'm leaning toward a collector that would poll all the > filers via SSH or SDK to run "options" and dump that into static files > on a normal export that the monitoring system could examine. Anyone have > any other suggestions? > > Thanks! > > ------------------------------------------------------------------------------------- > > A Darren Dunham <mailto:ddunham@taos.com> > Donnerstag, 7. April 2011 23:09 > > > I would like to check the options on my filers for various items, but > I'm not sure the best way to do it. Could anyone tell me if I'm missing > something? > > SNMP and Operations Manager: I don't see that I can pull the data from > either of these sources. > > SSH and SDK access: I could create an account, but it seems to me that the only > restriction available is the ability to run the "options" command. > There's no way to grant an account read-only access to the data. I'd > rather not give the monitor direct access to modify things. > > NFS: The current options appear to be visible as a local file in /etc, > but creating NFS access to vol0 for lots of filers for a single file > seems more of a hassle to me. > > At the moment I'm leaning toward a collector that would poll all the > filers via SSH or SDK to run "options" and dump that into static files > on a normal export that the monitoring system could examine. Anyone have > any other suggestions? > > Thanks! > > ------------------------------------------------------------------------------------- Hello Toasters, I am in the process of moving our data to new shelfs via snapmirror. In my last test I swichted to the new vols, rebooted and realized that our iSCSI settings (initiator groups, mappings) where missing. Where are they stored? Is there something I forgot? Is there any Document I should read? Thanks from Hamburg, René -- ----------------------------------------- Telefonkonferenzen zum Festnetzpreis! Einfach kostenlos Konferenzraum anfordern http://www.talkyoo.net ----------------------------------------- René Bormann Geschäftsführer BB Tel GmbH Jungfernstieg 30, 20354 Hamburg Register: Amtsgericht Hamburg, HRB 86074 Geschäftführer: René Bormann Tel: +49 (0) 1805 - 65 777 777 * Fax: +49 (0) 1805 - 65 777 799 * http://www.bbtel.de ----------------------------------------- * (bbtel.de 14ct/min) ** (es entstehen die Kosten, die ihre Telefongesellschaft zu einer Hamburger Festnetz-Rufnummer berechnet.) -----------------------------------------

2 1

iSCSI settings? where are they stored or backuped?
by René Bormann 11 Apr '11

11 Apr '11

Hello Toasters, I am in the process of moving our data to new shelfs via snapmirror. In my last test I swichted to the new vols, rebooted and realized that our iSCSI settings (initiator groups, mappings) where missing. Where are they stored? Is there something I forgot? Is there any Document I should read? Thanks from Hamburg, René -- ----------------------------------------- Telefonkonferenzen zum Festnetzpreis! Einfach kostenlos Konferenzraum anfordern http://www.talkyoo.net ----------------------------------------- René Bormann Geschäftsführer BB Tel GmbH Jungfernstieg 30, 20354 Hamburg Register: Amtsgericht Hamburg, HRB 86074 Geschäftführer: René Bormann Tel: +49 (0) 1805 - 65 777 777 * Fax: +49 (0) 1805 - 65 777 799 * http://www.bbtel.de ----------------------------------------- * (bbtel.de 14ct/min) ** (es entstehen die Kosten, die ihre Telefongesellschaft zu einer Hamburger Festnetz-Rufnummer berechnet.) -----------------------------------------

1 0

Easy way to monitor options?
by A Darren Dunham 08 Apr '11

08 Apr '11

I would like to check the options on my filers for various items, but I'm not sure the best way to do it. Could anyone tell me if I'm missing something? SNMP and Operations Manager: I don't see that I can pull the data from either of these sources. SSH and SDK access: I could create an account, but it seems to me that the only restriction available is the ability to run the "options" command. There's no way to grant an account read-only access to the data. I'd rather not give the monitor direct access to modify things. NFS: The current options appear to be visible as a local file in /etc, but creating NFS access to vol0 for lots of filers for a single file seems more of a hassle to me. At the moment I'm leaning toward a collector that would poll all the filers via SSH or SDK to run "options" and dump that into static files on a normal export that the monitoring system could examine. Anyone have any other suggestions? Thanks! -- Darren

2 2

Re: Easy way to monitor options?
by A Darren Dunham 08 Apr '11

08 Apr '11

On Fri, Apr 08, 2011 at 04:27:44PM +0200, Jan-Pieter Cornet wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Actually, if you give an account just the capabilities "login-ssh,cli-options", then that account will ONLY be able to run the 'options' command (which will list all current option settings), and it will not be able to change any options. That was what I tried first, but it doesn't seem to work. # ssh testfiler useradmin user list monitor Name: monitor Info: Rid: 131075 Groups: Monitors Full Name: Allowed Capabilities: login-ssh,cli-options Password min/max age in days: 0/4294967295 Status: enabled # ssh monitor@testfiler options autologout.telnet.timeout monitor@testfiler's password: autologout.telnet.timeout 60 # ssh monitor@testfiler options autologout.telnet.timeout 30 monitor@testfiler's password: # ssh monitor@testfiler options autologout.telnet.timeout monitor@testfiler's password: autologout.telnet.timeout 30 So with just those two privs, it can change settings. > > Likewise with the SDK, you can enable specific functions, in your case api-options-list-info and possibly api-options-get. Hmm. I didn't realize the different privs available there. Even though I have DFM and will probably use it for this task, I'm very interested in knowing how the SDK could enable this to happen. Thanks! -- Darren

1 0

Distribute aggregate across shelves or limit to one shelf?
by Ray Van Dolson 08 Apr '11

08 Apr '11

Best practice (based on my reading of the archives) seems to be to distribute disk membership in an aggregate across disk shelves. This would appear to be for performance reasons primarily (less chance of saturating a shelf's "uplink" to the controller), but how does it affect reliability? If I limit myself to one aggregate per shelf, if I lose that shelf I lose only the one aggregate. If aggregates are distributed I could lose all of them. My thought is that the chance of the shelf failing is actually pretty slim as its hardware isn't all that sophisticated. And obviously there are performance penalties for limiting to one aggregate per shelf (disk count maximums). Thanks, Ray

5 4

RE: Easy way to monitor options?
by Schneidewind, Scott 08 Apr '11

08 Apr '11

You should talk with your NetApp account team. Most of the new systems include Operations Manager (DFM) software at no charge. From: Antonio Varni [mailto:avarni@estalea.com] Sent: Thursday, April 07, 2011 11:34 PM To: Schneidewind, Scott Cc: A Darren Dunham; toasters(a)mathworks.com Subject: Re: Easy way to monitor options? good advice if you are licensed for DFM

1 0

DS14 (LRC) - ESH2 Modules?
by René Bormann 08 Apr '11

08 Apr '11

Hello all, we got some old DS14 Shelfs with LRC Modules. Is it possible to replace the LRC Modules with ESH2? Thanks from Hamburg, René -- ----------------------------------------- Telefonkonferenzen zum Festnetzpreis! Einfach kostenlos Konferenzraum anfordern http://www.talkyoo.net ----------------------------------------- René Bormann Geschäftsführer BB Tel GmbH Jungfernstieg 30, 20354 Hamburg Register: Amtsgericht Hamburg, HRB 86074 Geschäftführer: René Bormann Tel: +49 (0) 1805 - 65 777 777 * Fax: +49 (0) 1805 - 65 777 799 * http://www.bbtel.de ----------------------------------------- * (bbtel.de 14ct/min) ** (es entstehen die Kosten, die ihre Telefongesellschaft zu einer Hamburger Festnetz-Rufnummer berechnet.) -----------------------------------------

5 5

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters