A small datacenter build and other projects

[just2cool]

infiniband really??? WHY!?!?!?!?!?!?!?

We also use infiniband extensively. It is used in every serious wall street / financial firm in the world like mine, as well as a lot of HPC environments.

We don't care about the cost savings. The latency difference (especially due to RDMA) is huge. RDMAoE/iWARP isn't quite there yet.

BTW -- nice stuff spyder. I've been looking at this thread from time to time. You seem to be a jack of all trades.

 

[cactus]

I typed a huge response to this on another forum. This is the short version.

Cost of just a single 10GB 24 port switch > All cabling/cards/36port for a IB deployment at 40Gbs for our cluster.

10GB has not reached an affordable price yet.

Are you going to be using IPoIB, SDP, RDMA, eIPoIB, or etc? Strictly IB network or GbE for something other than management?

 

[The Spyder]

We run IPoIB right now. There is a separate gigabit network for IPMI. It works great for our application.

 

[The Spyder]

We also use infiniband extensively. It is used in every serious wall street / financial firm in the world like mine, as well as a lot of HPC environments.

We don't care about the cost savings. The latency difference (especially due to RDMA) is huge. RDMAoE/iWARP isn't quite there yet.

BTW -- nice stuff spyder. I've been looking at this thread from time to time. You seem to be a jack of all trades.

Thanks!

IB ended up giving us great speeds at a price we could not turn down. The longest distance we need is for the remote analyst stations, which are no more then 7m away.

I am never happy doing one repetitive task, hence the jack of all trades. I do find it difficult to properly advertise this when seeking employment.

 

[Electrofreak]

This thread makes me jealous. I work with a lot of really cool equipment (I manage enterprise equipment for an ISP), but it's all remote. Well, technically some of it is in the data center downstairs but I'm not a hardware guy so I only get to look through the glass...

 

[The Spyder]

Super small update:

We are starting testing on Phi integration to our existing cluster. Sadly, I have been spending all my time filling in for production. We just interviewed a fellow to fill the position, but we will see what happens. Otherwise I am still configuring quite a bit of this gear. In the next few months, phones, AC, and backup SAN's will be ordered. At the same time- I am getting ready to start eval on new node/controller replacements and storage replacements for our next gen clusters. Hopefully I won't be too bored, haha!

Fun question: Are you running a collapsed core network? If so, have any feedback? I am considering redoing a bit of my networking and setting up two e5406zl chassis up with MSTP and VRRP in a collapsed core setup for the DC. Fun times!

My users are eagerly waiting the new VPN/Site to site link that is FINALLY being deployed in two weeks. Should be a nice speed increase.

 

[calvinj]

Most of the places I've been at are small enough to have a collapsed core. It worked all right, no complaints here.

 

[nry]

We run IPoIB right now. There is a separate gigabit network for IPMI. It works great for our application.

What sort of performance do you see with IPoIB? Been considering this myself but unsure at the moment due to the amount of bad posts I have read about only getting 1.5Gbps bandwidth with really high CPU usage!


Would love to know more about that Phi coprocessor, not seen them before

 

[The Spyder]

We see 3.2Gb's just on Windows 7 over CIFS. It's awesome.

 

[nry]

Cool still not sure if I should go down the Infiniband route or try find some bearable prices on 10g copper!

 

[Deleted member 88227]

Now that's the kind of job I'd like to have. Screw my current job.

 

[The Spyder]

Another small update:

Updating a bulk lot of Crucial M4's

Production tech managed to drop his laptop off my IT cart, on the screen...

Our main server room has been a mess- and honestly still is. I re-did a bunch of cabling and removed some dead UPS's.

My tech labeled all the cables and it's now manageable. I was out of short cables, ignore the bundles.

Core all cleaned up and ready to be swapped out- more on this later.

Rack cleaned up.

I broke my 24" and borrowed a 2nd 30"- Loving it.

Night shot- seeing as we left at 8pm.

Coming up in the next few weeks I will be swapping firewalls, installing a new rack in the main server room, and setting up AD. After that I am hoping to rebuild the datacenter network in a collapsed core model. More funding becomes available after March and then we will start purchasing backup hardware/phones/ AC.

 

[techtips]

Nice happy face in the 2nd last pic

Knowing myself I would have had the banana facing a different way lol

 

[D-EJ915]

Production tech managed to drop his laptop off my IT cart, on the screen...

Our main server room has been a mess- and honestly still is. I re-did a bunch of cabling and removed some dead UPS's.

I guess you can't blame your guy for breaking it when yours is precariously placed as well lol.

 

[The Spyder]

Haha, funny story: I have nerve damage in my left hand. I picked up that MBA and was talking with my boss/walking out towards our ware house. I ended up clipping the wall with my wrist, pinching a nerve causing my grip to release. My other hand flew over and pinned the MBA against the wall, about 2ft from the floor. Has a nice 3" scratch on it and a blemished corner. It was apparently quite the site- and my boss now understands why I order everything with accidental damage .

 

[extide]

Xeon Phi!! I am so jealous!! Can you please see if they can run Folding@HOME and if so can you get some performance numbers?

 

[The Spyder]

Another small update.


I decided to change my network design at our main office. I moved the 5406 chassis +10gb card to our production building for a collapsed core design. Replacing it is a 2910al + cold spare backup. It's working great so far.

This weekend I added a few more runs of Cat6 for the internet lines. I relocated our router from a storage closet to the AC'd server room, but left the modems in there.

Rainbow of patch cords in the storage room.

We did some port matching (patch->switch) for VLAN assignment, sadly it's a bit messy now. I need to remove the barracuda and clean up when I have a chance.

The back of the ASA.

Here's hoping we get some fun projects in the future .

 

[nry]

What network leads/boots are they? Look much better than the standard crap I use

 

[jeffmoss26]

Looks like Ortronics to me.

 

[The Spyder]

Allen Tel.

 

[jeffmoss26]

D'oh! I should know, Graybar is just down the street from my office

 

[The Spyder]

We finished up the server room a few nights back. In a few months I will have a new phone server and backup NAS to install- once that is done, I can remove our main storage and move it to the DC. Our site to site link is working decently, but our internet is killing me. Comcast Business - modem is locking 1-2 times a week. I'm thinking of upgrading to their fiber service. The SLA on our site to site link has been great. No unscheduled downtime.

Got my MiniGooseII reinstalled.

I still need to order some shorter cables.

The switches/ASA are all connected via PDU/battery backup now. Much better.

Most of our time has been spent getting Nagios/Cacti installed and everything setup. Can't wait to get a small screen to hang in the office + a raspberry pi. Funding is still.... on hold for this year. I can't wait to get backups/AC/ new phone system going.

 

[calvinj]

Couple of Things..

Nagios + MRTG.. Know it sounds crazy, but thats what we have here and it works rather well. Can monitor the mrtg file and then alert against it during peak usage if you wanted too. Not sure you can do that with cacti. Nagios is a PITA as far as it goes for configuration.

Those HP Switches still treating you well? Been looking at some here as well.

 

[The Spyder]

I will check it out. The HP gear has been decent so far, no complaints outside of wishing I had researched some of the features better- but at the time I had not intended for them to be used as I want now. The chassis (54xx) are great.

 

[The Spyder]

I found a TV and brought one of my spare Raspberry Pi's in- working great so far. I need to get a 1080p tv though.

(10% cap map, hence the 456% internet, haha)

 

[green91]

Looks like my work -- Logitech MK550, Vsphere, & Cacti!

I do love Cacti. I've got an extensive monitoring system set up with Cacti, if you need any tips I'd be happy to share.

I broke my 24" and borrowed a 2nd 30"- Loving it.

.

 

[stryder2720]

Hot damn, you should have marked this thread "NSFW"

 

[p3n]

You realise cardboard doesn't conduct electricity right?

 

[The Spyder]

Yes, but my tech was being silly and insisted.

 

[Red Squirrel]

Actually the outer layer of an antistatic bag DOES conduct electricity, I'd actually remove it, or is it inside out? I never confirmed this for myself though, I just read it somewhere, so I could be wrong.

So are you using the Pi as a monitoring station? That's not a bad idea. Saves from using up a workstation screen, and is very low power. You can also get cases for the Pi: http://www.canakit.com/raspberry-pi-case-enclosure-box-black.html

I have pi here at home that I'm not sure what to use for yet. Debating between HTPC or environmental monitoring device (with an Arduino mega). Might buy another, and do both! These are so versatile and low cost, it's awesome.

 

[calvinj]

I will check it out. The HP gear has been decent so far, no complaints outside of wishing I had researched some of the features better- but at the time I had not intended for them to be used as I want now. The chassis (54xx) are great.

What did you wish they would do that they can't?

I found a TV and brought one of my spare Raspberry Pi's in- working great so far. I need to get a 1080p tv though.

How are you doing the heat maps? I would love to get that setup along with Nagios

 

[p3n]

Looks like PHP weathermap, presumably as a plugin for cacti. Currently playing with it myself.

 

[extide]

Actually the outer layer of an antistatic bag DOES conduct electricity, I'd actually remove it, or is it inside out? I never confirmed this for myself though, I just read it somewhere, so I could be wrong.

So are you using the Pi as a monitoring station? That's not a bad idea. Saves from using up a workstation screen, and is very low power. You can also get cases for the Pi: http://www.canakit.com/raspberry-pi-case-enclosure-box-black.html

I have pi here at home that I'm not sure what to use for yet. Debating between HTPC or environmental monitoring device (with an Arduino mega). Might buy another, and do both! These are so versatile and low cost, it's awesome.

They do conduct electricity, if they were an insulator it would allow large charges to be built up in a small place. As a conductor it spreads any charge out over the largest area. You can actually wrap computer parts in tin foil for the same/better effect as an anti static bag. Static bags do have some resistance, though. But yeah most people, incorrectly, think they are insulators. If they were, they wouldn't be able to do their job!

 

[obrith]

What did you wish they would do that they can't?



How are you doing the heat maps? I would love to get that setup along with Nagios

We've got Nagvis on top of our Icinga and it does heat mapping.

 

[The Spyder]

It's been too long- bump! Things have been... interesting around here- but sadly less so in the tech way. Would you believe it if I said I still do not have a AD environment fully setup? It's my goal over the next few weeks. Our new storage also has been slowly showing up. We underwent a pretty major office shift that has had me relocating the R+D sil and tons of users. Can't say I am bored, but it's keeping my away from the important projects.


In no particular order:

Got a free Foundry switch to play on, after 5 minutes of research finding nothing but complaints I now understand why it was free.

Moved our R+D 144TB NAS.

Moved our main storage from the corp office to the warehouse DC.

My tech after putting in the rails.

Moved R+D servers in to the now empty rack at the corp office.

It will never be this clean again, haha.

This is NOT how to ship servers!

The dance club downstairs keeps knocking my AP's loose- time for some Blue locktite.

Cleaning around the office during the moves.

We bought and install 50 SSD's in to new laptops last year, leaving us with 50 500gb 7200rpm 2.5" drives. Off to ebay they go!

Spent a night installing the last floor anchors before moving the main NAS.

Added 24" 1TB drives to my colo'd ESXi all-in-one.

 

[Red Squirrel]

Woah sexy, that is a lot of storage.

 

[calvinj]

I was just thinking about this thread today. About time!

 

[RiDDLeRThC]

Why didn't you use the mounting plates for the APs? I don't think anything would bring ours down.

 

[criccio]

I haven't posted in [H] in years and what brought me back was this thread = realizing you are a fellow rotard!

 

[/usr/home]

Why didn't you use the mounting plates for the APs? I don't think anything would bring ours down.

Yeah, the included brackets work great.