A small datacenter build and other projects

Haha, thanks guys.

And yes, I am a bit rotard'd. I'm building a new hobby shop to support the addition- check out FD3S Engineering on Facebook if you want to see the progress and cars I build.
 
Spyder,

Do you happen to have a updated list of stuff for sale, or a link to your ebay store?
 
Some tiny updates:

Check out my "ghetto" hot/cold isle. We used left over curtain from the warehouse and bought some extension ducts for the MovinCool26. It already dropped the temps 10f and once the new storage boxes are racked up, there should be no holes. I can't put a ceiling on it though.
2HnRuy3l.jpg

CyM8IUWl.jpg

mrcGCSTl.jpg

10pLeTHl.jpg


This week we built some of our new in house storage. (1) Is 22x240gb Intel 520 series and the other two are 24x3TB.
ZEQh7x4l.jpg
 
The Spyder said:
Yes, Solaris 11 and Napp-It.
Click to expand...

Any experience using FreeNAS/TrueNAS in your deployment? I am thinking of FreeBSD vs illumos/napp-it variants for a Radiology image storage server (not a lot of IOps, but availability and scalability is key).
 
Last edited:
For us, the lack of infiniband support killed freenas and truenas licensing prevented us from using any existing hardware- so outside of demo's- no. We have just over 4PB total. Our basic station ships with 336TB raw and we add 168TB JBODS any time we need to expand. Adding them is as simple as connecting to the HBA, adding them to the pool and you're good to go. Setting up duplication between systems is pretty simple.
 
The Spyder said:
For us, the lack of infiniband support killed freenas and truenas licensing prevented us from using any existing hardware- so outside of demo's- no. We have just over 4PB total. Our basic station ships with 336TB raw and we add 168TB JBODS any time we need to expand. Adding them is as simple as connecting to the HBA, adding them to the pool and you're good to go. Setting up duplication between systems is pretty simple.
Click to expand...


Thanks for the insight! :)
 
@The Spyder

Did you ever consider NexentaStor for your ZFS storage management? If so, what were your reasons for rejecting it? Cost? Performance?
 
Spyder. I just sat here and read this entire thread. You sir are doing great things. Keep it up! I hope one day when I get to run the IT show here where I work I can do just as good of a job. Thanks for the inspiration bud.
 
parityboy said:
@The Spyder

Did you ever consider NexentaStor for your ZFS storage management? If so, what were your reasons for rejecting it? Cost? Performance?
Click to expand...

Our original datastation was based off it, back when there was no 18tb limit. We just recently had to look at replacements and due to things outside our controls, we were only allowed three choices of OS. As far as my in house storage goes, Nexanta, while being very inexpensive compared to the other big players, just is extra overhead. Solaris/Omni/OI with Napp-It has been stable and reliable. In the future I would love to have EMC/Netapp for my ESXi hosts, but for now, it works.
 
EspoNation said:
Spyder. I just sat here and read this entire thread. You sir are doing great things. Keep it up! I hope one day when I get to run the IT show here where I work I can do just as good of a job. Thanks for the inspiration bud.
Click to expand...

Glad you enjoyed the read! It's been a interesting journey so far. I should really update the thread with the latest progress.
 
It's been busy again around here. I am happy to say, I finally have AC for our warehouse DC. My storage project is moving along and hopefully later this fall I can order some new VM hosts to replace our 5yr old supermicro white boxes.

Let's start with a backplane change out- this was a PITA- the new chassis came with a single expander and I switched to a non-expander
RuaABFjl.jpg

xdT6o0ol.jpg

eh4fiCil.jpg

vao0dpSl.jpg


I could hear this as soon as I walked into the warehouse.
ENRF8fGl.jpg


Awesome dev system. Anyone guess what it is?
sy8gLkSl.jpg


Inside units placed.
Rwf1znNl.jpg


Crane service for the AC units- keeping it classy.
0GSzzl2l.jpg

rW4Z1Bsl.jpg

pFircwGl.jpg


Our new production guy doesn't know his own strength.
hePCHvZl.jpg


More AC progress.
hGbXbTbl.jpg

eI0oxm7l.jpg

O5scXs8l.jpg


Condensate pump and power wrapped up.
lStCInXl.jpg


Return vents completed.
xC7lAlUl.jpg


Thermostat
DdUv6jql.jpg


Ouside air intake duct on the left.
Cq2vRQLl.jpg


View of the ducting from the other side.
SzYWDbdl.jpg


External air economizer temp control.
xo2DnGXl.jpg


By keeping the thermostats set at a ~75 temp, the units cycle nearly half as much. This was later set to 75. Ignore the 80 :).
UXXNlAIl.jpg


Supermicro 2bay rear 2.5" drive tray.
bqWjN7Zl.jpg


Drives Installed
2ilZyxgl.jpg


All buttoned up.
4aLeqWPl.jpg
 
Last edited:
What are those other 3 expansion cards in the storage box? Some sort of APU? No idea what that dev board is but seems to have every connector under the sun!
 
I want your job.... Enough said.


Looks like you have enough AC power there to keep that 68 year around :D
 
Nice stuff!

Funny, I've been planing out my cooling solution for my own server room at home and it will also be intaking air from outside depending on weather. I just hit a set back though, my system was going to be based on a series of electric dampers controlled by relays that open the proper ones based on the inside/outside temp but it seems those are not very easy to find and the few places I found that sell em want like 600 bucks a pop.
 
Red Squirrel said:
Nice stuff!

Funny, I've been planing out my cooling solution for my own server room at home and it will also be intaking air from outside depending on weather. I just hit a set back though, my system was going to be based on a series of electric dampers controlled by relays that open the proper ones based on the inside/outside temp but it seems those are not very easy to find and the few places I found that sell em want like 600 bucks a pop.
Click to expand...

Try this place. I see quite a few used relays running through there from time to time.

http://stores.ebay.com/liquishightechequipment
 
The Spyder said:
Our original datastation was based off it, back when there was no 18tb limit. We just recently had to look at replacements and due to things outside our controls, we were only allowed three choices of OS. As far as my in house storage goes, Nexanta, while being very inexpensive compared to the other big players, just is extra overhead. Solaris/Omni/OI with Napp-It has been stable and reliable. In the future I would love to have EMC/Netapp for my ESXi hosts, but for now, it works.
Click to expand...

With 4PB, I'm sure EMC/NetApp would love to have your business as well.
 
Today we finished up installing 144TB of spindle storage and 5.7TB of SSD storage for each location. The AC has been amazing, keeping the warehouse nice a mild during our 90 degree weather. (I realize this is rather mild compared to elsewhere, haha). Hopefully I can order some replacement hosts for my VM infrastructure soon, but we shall see. I've been pondering the question of to white box or to not. Dell Poweredge R720's are dollar for dollar identical to the Supermicro configs I have looked at. On the flip side, I would LOVE a set of Cisco UCS blades. I guess the budget will tell :).
 
If you must go rack mounts and you can't get Cisco blades, look at their C Series stuff. I was able to get it cheaper than my R720s.
 
QHalo said:
If you must go rack mounts and you can't get Cisco blades, look at their C Series stuff. I was able to get it cheaper than my R720s.
Click to expand...

Huh? UCS chassis are rack mountable.

From an infrastructure management standpoint UCS is the the way to go. Best investment our company has ever made.
 
Awesome dev system. Anyone guess what it is?
sy8gLkSl.jpg


looks like a advantech SOD-DB5800.
What are you using it for?
they are a interesting peace of tech...
i used a similar Computer on Module system in a "extremely" remote site.

keep up the good work
 
Internal R+D project I can't speak about. We may or may not use them due to some issues we have come across. Neat little boards.
 
Seem's there is a rather lack of updates on my behalf, sorry about that. With our second restructure, things have really quieted down and frankly I am not sure what's going to happen. While things have settled, I'm having a hard time keeping my tech busy. Not trying to rant, but it's the downside to all the awesome hardware/systems here.

We did replace our Unifi Pro AP's with AC ones- it fixed our problem with Apple clients dropping.
 
I just spent the best part of a couple of hours reading this thread, awesome tech, how do you find the SuperMicros in production? solid? many failures? Also on the subject of cars, I have a weekend warrior Subaru STI with 400bhp, its particularly awesome in the snow; that is if you remember all that awesome grip comes when you apply power, as soon as you let off its basically a sledge unless you have studs or chains :D
 
Someone has the most epic job! Building a small datacenter and keeping it alive (+ all the other stuff you're doing)... And all those portable units... :drool:

Very nice! Must be so much nicer to have the freedom compared to a bigger datacenter environment.
 
I had a site in the middle of Kenya and another at the top of a mountain in Chilli, that's remote! :D
 
Thanks for the continued support. I won't rant about the company I work for, but let me put it this way- some less then popular decisions have been made recently by the upper management and no one is happy- including myself. Things are rocky in the contract world and we sadly laid several people off due to refocusing our efforts, including my tech.

Some good news:
A handful of projects has been approved and I can resume posting cool stuff like this:
2LqNuWE.jpg


It's going to be a interesting year- I can promise that.
 
I am so happy this thread keeps popping up. I even take time when I am slow and find it again to re read the entire thing.
 
The Spyder said:
Some good news:
A handful of projects has been approved and I can resume posting cool stuff like this:
2LqNuWE.jpg


It's going to be a interesting year- I can promise that.
Click to expand...

Hmm homebrew GPU compute node or maybe some sort of GPU offload for VDI? Been foaming at the mouth for GPU offload for VDI for awhile. Would love to slice and dice our engineering boxes into VDI clients that can take resources as they need it.
 
Awesome stuff Spyder. I was an end-user of your company's service last year in the desert. I always wondered what hardware was used for the processing and storage and it's more impressive than I thought. Nice to see the evolution from a few dozen TB to the PB level.
 
You must log in or register to reply here.
Back
Top