A small datacenter build and other projects

pug said:
infiniband really??? WHY!?!?!?!?!?!?!?
Click to expand...

We also use infiniband extensively. It is used in every serious wall street / financial firm in the world like mine, as well as a lot of HPC environments.

We don't care about the cost savings. The latency difference (especially due to RDMA) is huge. RDMAoE/iWARP isn't quite there yet.

BTW -- nice stuff spyder. I've been looking at this thread from time to time. You seem to be a jack of all trades.
 
The Spyder said:
I typed a huge response to this on another forum. This is the short version.

Cost of just a single 10GB 24 port switch > All cabling/cards/36port for a IB deployment at 40Gbs for our cluster.

10GB has not reached an affordable price yet.
Click to expand...

Are you going to be using IPoIB, SDP, RDMA, eIPoIB, or etc? Strictly IB network or GbE for something other than management?
 
We run IPoIB right now. There is a separate gigabit network for IPMI. It works great for our application.
 
just2cool said:
We also use infiniband extensively. It is used in every serious wall street / financial firm in the world like mine, as well as a lot of HPC environments.

We don't care about the cost savings. The latency difference (especially due to RDMA) is huge. RDMAoE/iWARP isn't quite there yet.

BTW -- nice stuff spyder. I've been looking at this thread from time to time. You seem to be a jack of all trades.
Click to expand...

Thanks!

IB ended up giving us great speeds at a price we could not turn down. The longest distance we need is for the remote analyst stations, which are no more then 7m away.

I am never happy doing one repetitive task, hence the jack of all trades. I do find it difficult to properly advertise this when seeking employment.
 
This thread makes me jealous. I work with a lot of really cool equipment (I manage enterprise equipment for an ISP), but it's all remote. Well, technically some of it is in the data center downstairs but I'm not a hardware guy so I only get to look through the glass... :(
 
Super small update:

na4XWzm.jpg


We are starting testing on Phi integration to our existing cluster. Sadly, I have been spending all my time filling in for production. We just interviewed a fellow to fill the position, but we will see what happens. Otherwise I am still configuring quite a bit of this gear. In the next few months, phones, AC, and backup SAN's will be ordered. At the same time- I am getting ready to start eval on new node/controller replacements and storage replacements for our next gen clusters. :) Hopefully I won't be too bored, haha!

Fun question: Are you running a collapsed core network? If so, have any feedback? I am considering redoing a bit of my networking and setting up two e5406zl chassis up with MSTP and VRRP in a collapsed core setup for the DC. Fun times!

My users are eagerly waiting the new VPN/Site to site link that is FINALLY being deployed in two weeks. Should be a nice speed increase.
 
Most of the places I've been at are small enough to have a collapsed core. It worked all right, no complaints here.
 
The Spyder said:
We run IPoIB right now. There is a separate gigabit network for IPMI. It works great for our application.
Click to expand...

What sort of performance do you see with IPoIB? Been considering this myself but unsure at the moment due to the amount of bad posts I have read about only getting 1.5Gbps bandwidth with really high CPU usage!


Would love to know more about that Phi coprocessor, not seen them before :D
 
Cool still not sure if I should go down the Infiniband route or try find some bearable prices on 10g copper!
 
Now that's the kind of job I'd like to have. Screw my current job.
 
Another small update:

Updating a bulk lot of Crucial M4's
8vVQAyjl.jpg


Production tech managed to drop his laptop off my IT cart, on the screen...
kC6mzvKl.jpg


Our main server room has been a mess- and honestly still is. I re-did a bunch of cabling and removed some dead UPS's.
TAM41pTl.jpg

HpGF2zPl.jpg


My tech labeled all the cables and it's now manageable. I was out of short cables, ignore the bundles.
JAavijul.jpg

3DDgAYsl.jpg


Core all cleaned up and ready to be swapped out- more on this later.
VQ9hMk3l.jpg


Rack cleaned up.
eXkXBYil.jpg


I broke my 24" and borrowed a 2nd 30"- Loving it.
Z6wOg0tl.jpg


Night shot- seeing as we left at 8pm.
8EfQR7Rl.jpg



Coming up in the next few weeks I will be swapping firewalls, installing a new rack in the main server room, and setting up AD. After that I am hoping to rebuild the datacenter network in a collapsed core model. More funding becomes available after March and then we will start purchasing backup hardware/phones/ AC.
 
Nice happy face in the 2nd last pic :p

Knowing myself I would have had the banana facing a different way lol
 
The Spyder said:
Production tech managed to drop his laptop off my IT cart, on the screen...
kC6mzvKl.jpg


Our main server room has been a mess- and honestly still is. I re-did a bunch of cabling and removed some dead UPS's.
TAM41pTl.jpg
Click to expand...
I guess you can't blame your guy for breaking it when yours is precariously placed as well lol.
 
Haha, funny story: I have nerve damage in my left hand. I picked up that MBA and was talking with my boss/walking out towards our ware house. I ended up clipping the wall with my wrist, pinching a nerve causing my grip to release. My other hand flew over and pinned the MBA against the wall, about 2ft from the floor. Has a nice 3" scratch on it and a blemished corner. It was apparently quite the site- and my boss now understands why I order everything with accidental damage :D.
 
Xeon Phi!! I am so jealous!! Can you please see if they can run Folding@HOME and if so can you get some performance numbers?
 
Another small update.


I decided to change my network design at our main office. I moved the 5406 chassis +10gb card to our production building for a collapsed core design. Replacing it is a 2910al + cold spare backup. It's working great so far.
kURcw13l.jpg

VK9ohGJl.jpg


This weekend I added a few more runs of Cat6 for the internet lines. I relocated our router from a storage closet to the AC'd server room, but left the modems in there.

BFAZgxbl.jpg

Rainbow of patch cords in the storage room.
i7XC22al.jpg


We did some port matching (patch->switch) for VLAN assignment, sadly it's a bit messy now. I need to remove the barracuda and clean up when I have a chance.
JvOdx0Kl.jpg


The back of the ASA.
nb77U2yl.jpg



Here's hoping we get some fun projects in the future :).
 
What network leads/boots are they? Look much better than the standard crap I use
 
We finished up the server room a few nights back. In a few months I will have a new phone server and backup NAS to install- once that is done, I can remove our main storage and move it to the DC. Our site to site link is working decently, but our internet is killing me. Comcast Business - modem is locking 1-2 times a week. I'm thinking of upgrading to their fiber service. The SLA on our site to site link has been great. No unscheduled downtime.

uPyuDjyl.jpg


Got my MiniGooseII reinstalled.
NhaPMMml.jpg


I still need to order some shorter cables.
4vs48DQl.jpg


The switches/ASA are all connected via PDU/battery backup now. Much better.
inBUd14l.jpg




Most of our time has been spent getting Nagios/Cacti installed and everything setup. Can't wait to get a small screen to hang in the office + a raspberry pi. Funding is still.... on hold for this year. I can't wait to get backups/AC/ new phone system going. :)
 
Couple of Things..

Nagios + MRTG.. Know it sounds crazy, but thats what we have here and it works rather well. Can monitor the mrtg file and then alert against it during peak usage if you wanted too. Not sure you can do that with cacti. Nagios is a PITA as far as it goes for configuration.

Those HP Switches still treating you well? Been looking at some here as well.
 
I will check it out. The HP gear has been decent so far, no complaints outside of wishing I had researched some of the features better- but at the time I had not intended for them to be used as I want now. The chassis (54xx) are great.
 
I found a TV and brought one of my spare Raspberry Pi's in- working great so far. I need to get a 1080p tv though.

M3rVJbOl.jpg

(10% cap map, hence the 456% internet, haha)

HmmBeyLl.jpg
 
Looks like my work -- Logitech MK550, Vsphere, & Cacti!

I do love Cacti. I've got an extensive monitoring system set up with Cacti, if you need any tips I'd be happy to share.

The Spyder said:
I broke my 24" and borrowed a 2nd 30"- Loving it.
Z6wOg0tl.jpg

.
Click to expand...
 
Actually the outer layer of an antistatic bag DOES conduct electricity, I'd actually remove it, or is it inside out? I never confirmed this for myself though, I just read it somewhere, so I could be wrong.

So are you using the Pi as a monitoring station? That's not a bad idea. Saves from using up a workstation screen, and is very low power. You can also get cases for the Pi: http://www.canakit.com/raspberry-pi-case-enclosure-box-black.html

I have pi here at home that I'm not sure what to use for yet. Debating between HTPC or environmental monitoring device (with an Arduino mega). Might buy another, and do both! These are so versatile and low cost, it's awesome.
 
The Spyder said:
I will check it out. The HP gear has been decent so far, no complaints outside of wishing I had researched some of the features better- but at the time I had not intended for them to be used as I want now. The chassis (54xx) are great.
Click to expand...

What did you wish they would do that they can't?

The Spyder said:
I found a TV and brought one of my spare Raspberry Pi's in- working great so far. I need to get a 1080p tv though.
Click to expand...

How are you doing the heat maps? I would love to get that setup along with Nagios
 
Red Squirrel said:
Actually the outer layer of an antistatic bag DOES conduct electricity, I'd actually remove it, or is it inside out? I never confirmed this for myself though, I just read it somewhere, so I could be wrong.

So are you using the Pi as a monitoring station? That's not a bad idea. Saves from using up a workstation screen, and is very low power. You can also get cases for the Pi: http://www.canakit.com/raspberry-pi-case-enclosure-box-black.html

I have pi here at home that I'm not sure what to use for yet. Debating between HTPC or environmental monitoring device (with an Arduino mega). Might buy another, and do both! These are so versatile and low cost, it's awesome.
Click to expand...

They do conduct electricity, if they were an insulator it would allow large charges to be built up in a small place. As a conductor it spreads any charge out over the largest area. You can actually wrap computer parts in tin foil for the same/better effect as an anti static bag. Static bags do have some resistance, though. But yeah most people, incorrectly, think they are insulators. If they were, they wouldn't be able to do their job!
 
It's been too long- bump! Things have been... interesting around here- but sadly less so in the tech way. Would you believe it if I said I still do not have a AD environment fully setup? It's my goal over the next few weeks. Our new storage also has been slowly showing up. We underwent a pretty major office shift that has had me relocating the R+D sil and tons of users. Can't say I am bored, but it's keeping my away from the important projects.


In no particular order:

Got a free Foundry switch to play on, after 5 minutes of research finding nothing but complaints I now understand why it was free.
clgDn88l.jpg


Moved our R+D 144TB NAS.
DLMU1Tsl.jpg



Moved our main storage from the corp office to the warehouse DC.
wM3SBlkl.jpg

SuDe8t2l.jpg

My tech after putting in the rails.
r9AzVMkl.jpg

mDLR5HEl.jpg

MSuSL6ul.jpg

LtmOYTbl.jpg


Moved R+D servers in to the now empty rack at the corp office.
7XgbW0Al.jpg

It will never be this clean again, haha.
hgrXbHil.jpg


This is NOT how to ship servers!
Iw1RAwIl.jpg

aZERraul.jpg

jX5CxrNl.jpg

fVNEQKil.jpg


The dance club downstairs keeps knocking my AP's loose- time for some Blue locktite.
OmwdjT3l.jpg


Cleaning around the office during the moves.
uRvTw5Cl.jpg

zcXKYcul.jpg



We bought and install 50 SSD's in to new laptops last year, leaving us with 50 500gb 7200rpm 2.5" drives. Off to ebay they go!
ZC9RPhEl.jpg

Spent a night installing the last floor anchors before moving the main NAS.
DuD7IQEl.jpg


Added 24" 1TB drives to my colo'd ESXi all-in-one.
2iyT9r0l.jpg

dYsQ2dXl.jpg
 
Why didn't you use the mounting plates for the APs? I don't think anything would bring ours down.
 
I haven't posted in [H] in years and what brought me back was this thread = realizing you are a fellow rotard!
 
You must log in or register to reply here.
Back
Top