A small datacenter build and other projects

[MrGuvernment]

i still drewl over this thread

 

[The Spyder]

Haha, thanks guys.

And yes, I am a bit rotard'd. I'm building a new hobby shop to support the addition- check out FD3S Engineering on Facebook if you want to see the progress and cars I build.

 

[Das_Goose]

Spyder,

Do you happen to have a updated list of stuff for sale, or a link to your ebay store?

 

[The Spyder]

I do not ATM.

 

[The Spyder]

Some tiny updates:

Check out my "ghetto" hot/cold isle. We used left over curtain from the warehouse and bought some extension ducts for the MovinCool26. It already dropped the temps 10f and once the new storage boxes are racked up, there should be no holes. I can't put a ceiling on it though.

This week we built some of our new in house storage. (1) Is 22x240gb Intel 520 series and the other two are 24x3TB.

 

[surrealillusion]

Are you using some type of ZFS setup for your storage?

 

[The Spyder]

Yes, Solaris 11 and Napp-It.

 

[DICOMdude]

Yes, Solaris 11 and Napp-It.

Any experience using FreeNAS/TrueNAS in your deployment? I am thinking of FreeBSD vs illumos/napp-it variants for a Radiology image storage server (not a lot of IOps, but availability and scalability is key).

 

[The Spyder]

For us, the lack of infiniband support killed freenas and truenas licensing prevented us from using any existing hardware- so outside of demo's- no. We have just over 4PB total. Our basic station ships with 336TB raw and we add 168TB JBODS any time we need to expand. Adding them is as simple as connecting to the HBA, adding them to the pool and you're good to go. Setting up duplication between systems is pretty simple.

 

[DICOMdude]

For us, the lack of infiniband support killed freenas and truenas licensing prevented us from using any existing hardware- so outside of demo's- no. We have just over 4PB total. Our basic station ships with 336TB raw and we add 168TB JBODS any time we need to expand. Adding them is as simple as connecting to the HBA, adding them to the pool and you're good to go. Setting up duplication between systems is pretty simple.

Thanks for the insight!

 

[parityboy]

@The Spyder

Did you ever consider NexentaStor for your ZFS storage management? If so, what were your reasons for rejecting it? Cost? Performance?

 

[Deleted member 238539]

Spyder. I just sat here and read this entire thread. You sir are doing great things. Keep it up! I hope one day when I get to run the IT show here where I work I can do just as good of a job. Thanks for the inspiration bud.

 

[The Spyder]

@The Spyder

Did you ever consider NexentaStor for your ZFS storage management? If so, what were your reasons for rejecting it? Cost? Performance?

Our original datastation was based off it, back when there was no 18tb limit. We just recently had to look at replacements and due to things outside our controls, we were only allowed three choices of OS. As far as my in house storage goes, Nexanta, while being very inexpensive compared to the other big players, just is extra overhead. Solaris/Omni/OI with Napp-It has been stable and reliable. In the future I would love to have EMC/Netapp for my ESXi hosts, but for now, it works.

 

[The Spyder]

Spyder. I just sat here and read this entire thread. You sir are doing great things. Keep it up! I hope one day when I get to run the IT show here where I work I can do just as good of a job. Thanks for the inspiration bud.

Glad you enjoyed the read! It's been a interesting journey so far. I should really update the thread with the latest progress.

 

[The Spyder]

It's been busy again around here. I am happy to say, I finally have AC for our warehouse DC. My storage project is moving along and hopefully later this fall I can order some new VM hosts to replace our 5yr old supermicro white boxes.

Let's start with a backplane change out- this was a PITA- the new chassis came with a single expander and I switched to a non-expander

I could hear this as soon as I walked into the warehouse.

Awesome dev system. Anyone guess what it is?

Inside units placed.

Crane service for the AC units- keeping it classy.

Our new production guy doesn't know his own strength.

More AC progress.

Condensate pump and power wrapped up.

Return vents completed.

Thermostat

Ouside air intake duct on the left.

View of the ducting from the other side.

External air economizer temp control.

By keeping the thermostats set at a ~75 temp, the units cycle nearly half as much. This was later set to 75. Ignore the 80 .

Supermicro 2bay rear 2.5" drive tray.

Drives Installed

All buttoned up.

 

[p3n]

What are those other 3 expansion cards in the storage box? Some sort of APU? No idea what that dev board is but seems to have every connector under the sun!

 

[calvinj]

I want your job.... Enough said.


Looks like you have enough AC power there to keep that 68 year around

 

[Red Squirrel]

Nice stuff!

Funny, I've been planing out my cooling solution for my own server room at home and it will also be intaking air from outside depending on weather. I just hit a set back though, my system was going to be based on a series of electric dampers controlled by relays that open the proper ones based on the inside/outside temp but it seems those are not very easy to find and the few places I found that sell em want like 600 bucks a pop.

 

[MurphysLaww]

Nice stuff!

Funny, I've been planing out my cooling solution for my own server room at home and it will also be intaking air from outside depending on weather. I just hit a set back though, my system was going to be based on a series of electric dampers controlled by relays that open the proper ones based on the inside/outside temp but it seems those are not very easy to find and the few places I found that sell em want like 600 bucks a pop.

Try this place. I see quite a few used relays running through there from time to time.

http://stores.ebay.com/liquishightechequipment

 

[QHalo]

Our original datastation was based off it, back when there was no 18tb limit. We just recently had to look at replacements and due to things outside our controls, we were only allowed three choices of OS. As far as my in house storage goes, Nexanta, while being very inexpensive compared to the other big players, just is extra overhead. Solaris/Omni/OI with Napp-It has been stable and reliable. In the future I would love to have EMC/Netapp for my ESXi hosts, but for now, it works.

With 4PB, I'm sure EMC/NetApp would love to have your business as well.

 

[The Spyder]

Today we finished up installing 144TB of spindle storage and 5.7TB of SSD storage for each location. The AC has been amazing, keeping the warehouse nice a mild during our 90 degree weather. (I realize this is rather mild compared to elsewhere, haha). Hopefully I can order some replacement hosts for my VM infrastructure soon, but we shall see. I've been pondering the question of to white box or to not. Dell Poweredge R720's are dollar for dollar identical to the Supermicro configs I have looked at. On the flip side, I would LOVE a set of Cisco UCS blades. I guess the budget will tell .

 

[QHalo]

If you must go rack mounts and you can't get Cisco blades, look at their C Series stuff. I was able to get it cheaper than my R720s.

 

[schnell]

If you must go rack mounts and you can't get Cisco blades, look at their C Series stuff. I was able to get it cheaper than my R720s.

Huh? UCS chassis are rack mountable.

From an infrastructure management standpoint UCS is the the way to go. Best investment our company has ever made.

 

[QHalo]

You didn't comprehend what I wrote.

 

[SysOps]

Awesome dev system. Anyone guess what it is?

looks like a advantech SOD-DB5800.
What are you using it for?
they are a interesting peace of tech...
i used a similar Computer on Module system in a "extremely" remote site.

keep up the good work

 

[The Spyder]

Internal R+D project I can't speak about. We may or may not use them due to some issues we have come across. Neat little boards.

 

[The Spyder]

Seem's there is a rather lack of updates on my behalf, sorry about that. With our second restructure, things have really quieted down and frankly I am not sure what's going to happen. While things have settled, I'm having a hard time keeping my tech busy. Not trying to rant, but it's the downside to all the awesome hardware/systems here.

We did replace our Unifi Pro AP's with AC ones- it fixed our problem with Apple clients dropping.

 

[101001101]

I just spent the best part of a couple of hours reading this thread, awesome tech, how do you find the SuperMicros in production? solid? many failures? Also on the subject of cars, I have a weekend warrior Subaru STI with 400bhp, its particularly awesome in the snow; that is if you remember all that awesome grip comes when you apply power, as soon as you let off its basically a sledge unless you have studs or chains

 

[Das_Goose]

keep this alive!

 

[Yariko]

Someone has the most epic job! Building a small datacenter and keeping it alive (+ all the other stuff you're doing)... And all those portable units... :drool:

Very nice! Must be so much nicer to have the freedom compared to a bigger datacenter environment.

 

[calvinj]

Please tell me you have some updates you've been meaning to post?

 

[FLECOM]

"extremely" remote site.

satellite?

 

[Jay_2]

I had a site in the middle of Kenya and another at the top of a mountain in Chilli, that's remote!

 

[The Spyder]

Thanks for the continued support. I won't rant about the company I work for, but let me put it this way- some less then popular decisions have been made recently by the upper management and no one is happy- including myself. Things are rocky in the contract world and we sadly laid several people off due to refocusing our efforts, including my tech.

Some good news:
A handful of projects has been approved and I can resume posting cool stuff like this:

It's going to be a interesting year- I can promise that.

 

[Lames.]

Turning your datacenter into a mining center?

 

[Deleted member 238539]

I am so happy this thread keeps popping up. I even take time when I am slow and find it again to re read the entire thing.

 

[Hadrien]

Some good news:
A handful of projects has been approved and I can resume posting cool stuff like this:

It's going to be a interesting year- I can promise that.

Hmm homebrew GPU compute node or maybe some sort of GPU offload for VDI? Been foaming at the mouth for GPU offload for VDI for awhile. Would love to slice and dice our engineering boxes into VDI clients that can take resources as they need it.

 

[VisBits]

TT

This thread is sweet.

 

[Fenris_Ulf]

Awesome stuff Spyder. I was an end-user of your company's service last year in the desert. I always wondered what hardware was used for the processing and storage and it's more impressive than I thought. Nice to see the evolution from a few dozen TB to the PB level.

 

[techtips]

Any updates Spyder?