Official Red's server room thread

R

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
I posted most of these before here and there but realized I never made a dedicated thread anywhere or any forum for my server room project so decided to do it here.

I've also been thinking about making it into an actual "room" so figured I'd post what I got so far.

When I first moved into my house, this was my server room:



Not too impressive, and always made me nervous in case there is ever water in the basement. At some point I put them up on some small shelves.

I always wanted to get a rack but could not find one for a reasonable price. Then suddenly... I scored a 4 post rack for only a few hundred bucks online, new.



Around the same time I also scored 4 SAN enclosures so it was actually the first thing that went in, and the only thing rackmountable that I had at the time.








"Modem on a string" as opposed to "router on a stick" :p


Over the past years more stuff has gone into it, and more accessories around the house...








Full rack pic: Fairly recent. (sept 2012)



From top to bottom:

DIN rail with junction points for environmental control/monitoring around the house. This is basically just a patch panel. The second rail is for power distribution. Just keeps things clean.

Shelf with a relay controller and arduino board for the environmental stuff. The DIN rail has connections to this equipment.

Not seen, behind the shelf: Dell switch with patch panel.

Firewall: Running pfsense

hal9000: Environmental control server.

HDD dock for backups

borg: main server. This is pretty much the most important server I have. It does pretty much everything from file storage, email/spam filtering, web development, other misc development, VMs for various testing or other environments etc.... Been wanting to upgrade it. That is the same server that is in the black tower case with all the drives in the previous pics, I just moved it all to a rack mount.

McData fiber switch: not actually used

4x IBM SAN enclosures: not really used, but they are connected to the environmental server which has the appropriate fiber card. I use the top two enclosures for occasional backups of my whole environment.

Floor: UPSes. Small ones for less important stuff, and the black box and plastic container is my bigger UPS for the firewall, switch, WAP, and main server. It last's around 4-5 hours.

Closeup of DIN rails:




Environmental monitoring device:



Hvac control device:




big UPS: (inverter charger with batteries)




I've been wanting to actually enclose the server rack into a real room that is sealed off, fairly secure, and climate controlled. For climate control I will probably just have a large air filter and radiator with water pipes going throughout the floor as to disapate the heat to the rest of the house. Air will be forced from back of server room through this radiator then back in front. Might even seal it off in a way where air is forced to go through the rack. That means I could get away with less/no fans in some equipment. The server room does not generate much heat so I don't need a dedicated AC and rather not need one. For hydrogen potentially produced by the batteries I may need to have a separate system for that, probably just some small ABS pipe going outside and a small fan. I eventually want to have a battery shelf/enclosure and buy my own wire crimper so I can space them the way I want. There is not much selection of premade battery cables hence why the inverter is stuck against the batteries like that and can't even see the front display. I want to change that. A battery shelf will also let me stack more batteries together. I eventually want a rectifier/inverter setup, but inverter chargers are the most cost effective way of doing this and arn't as hard on the batteries as they're standby only.

This is another rack I scored, could not say no, it was free from work!




Not really sure what I will do with it TBH but it will most likely go beside my existing one. Probably what I'll end up doing is putting the new one beside the rack as is and use that one for lab stuff like if I decide to setup a Cisco lab or something. May also use it for power equipment if I get rackmount inverters and stuff. I thought of putting the batteries in it but it's kinda a waste to use a rack for that when I can just build a separate shelving system for those.

I just got my shares and been saving money, so I will probably be starting on the basement soon which will include finally enclosing the server room too. I will most likely have some kind of water based climate control and set it up so air is forced through the rack. I'll basically set it up as a "cold isle/hot isle" setup.
 
What gauge of battery cable and how long do you need? Pay shipping/materials and I will make you any length you want. I have 8, 4, 2, 00, and 0000 in the shop. Haha.
 
Not sure yet but I need to be able to make them on the fly so I will probably just buy the tools required.

Gauge is like 4 I think, but I'll probably go a bit bigger. They do get a tad warm and I'm only pulling around 350w. My plan is to build a shelf of sorts for the batteries to sit on, place them, and measure/cut and make them right there and then to the exact size I need. There's an electrical shop here so I think I can probably get them locally.

Though I will consider your offer if it turns out to be too expensive to buy the tools. What exactly do I need anyway, crimper and the ends? Does one crimper do multiple gauges? Can I just solder them in with a torch? I heard this can be done, but wonder how good it really is.
 
What do you have for power in the room?

20 amp? multiple 20 amps?

The only thing stopping me from doing a lot of what I want is power concerns in the locations.
 
Right now it's just a single 15 amp, but I plan to run a 100 amp sub panel there at some point. It will feed the rest of the basement too. I'm only using a fraction of a 15 amp circuit at this point though, but who knows what I could decide to add at some point. That SAN will use about 1000w when it's on so it's a decent chunk of power.
 
Going to be starting on this in the next few days once I'm done my night shifts. Here is a rough design for my battery rack for your viewing pleasure. This is just a draft. May add more reinforcement. The 2x4 main supports (red) will be bolted in the vertical runs (white) and the yellow will be either 1x6's or 2x6's. May also add more 2x4's between each shelf so the bolts alone arn't taking all the weight.



60cm by 80cm at the base, about 200cm high. The purple blocks represent batteries for size comparison.
 
Do be careful with that. Racking wet led-acid batteries that way in your basement is not to be trifled with. At least consider using sealed/gel batteries.
 
Yeah lot of precautions will be taken for sure. The batteries will also most likely sit in plastic containers like they are now. Basically adds a secondary layer in case of leak. The shelf will also be load tested to make sure it does not budge. Going to be anchored to the concrete floor and ceiling, too. Also need to add cross braces, not shown in the plan.

Also need to look into possibly hydrogen mitigation but I don't think that is really an issue as long as the charger is operating correctly. Hydrogen is produced when batteries are overcharged. hydrogen production = loosing electrolyte = batteries being slowly killed. I'll have a hydrogen sensor in that shelf system though, and it will be enclosed. Idealy I want to find some plastic panels to go in the front and back (with some venting up the top and bottom for heat) but not sure if I'll be able to find that somewhere.

The server room will also be hot/cold isle setup, so any vents I add will have a forced air flow of cooler air going through.

What I like about flooded lead acid is the ability to maintain them. I need to read up on that more, but basically I can add distilled water, or even special "reconditioners" to prolong their life. Need to read up more though so I know what indicators to look for to know when to do this. There's also different tools to test them, such as hydrometers.
 
for the price of sealed batteries, it really is a good investment. 120Ah worth can be had on ebay for $130 shipped.
 
More info on the batteries you're using? What charger are you using to charge them? How do you have them connected exactly? Basically like a off the shelf UPS. The equipment is connected directly to the batteries, but power is fed from the wall, in the event of a power failure the batteries take on the load? What equipment are you using to accomplish this?
 
Flip your AP over. The general radiation pattern of those UniFi's is like a donut and they radiate upwards from the base of the AP. You're focusing your signal towards your basement floor.
 
Red,

After spending 20+ years in the Telco business I think I've seen it all with battery issues. All the bad stuff....acid spills, fires, hydrogen explosions, etc. Even in places that took all the right precautions. Lots of good stuff too, including all the reasons why most large Telco offices still prefer wet batteries (and the $millions spent to make them safe). I'm also painfully familiar with the reasons they prefer sealed batteries in small spaces (cell sites, underground vaults, etc). All I can do is offer advice. You can take it or leave it as you will.

What you are doing - racking wet lead/acid automobile batteries onto a tall rack in a residential environment - really is dangerous. Please, for you and your family, take this seriously. This is not a minor discussion of 'code violations' pulling network cable in your house with well known/understood risks. You are playing with fire (quite literally).

For what its worth - your call to pay head:

- PLEASE use sealed batteries rated for data center use (like the ones APC sources). Shopping carefully you'll find they are not really that much more expensive than auto batteries.
- If you choose to use wet batteries, don't rack them over mid-chest height (about 4 feet) to avoid spills
- DONT enclose them. Make sure the room in well ventilated. A few PPM of H in the room is no worry, but you'll get higher concentrations in any enclosure you build
- Know that they WILL outgas - even if your charger is working correctly. Slowly, but they will.
- Know that electronics do fail. All electronics eventually fail. Plan assuming your charge circuit may fail and you will get a large outgas. Be sure your ventilation can clear it quickly.
- Again - PLEASE - for your own safety and your family - use sealed batteries.
 
My god, this scares the living shite out of me and its not even my house!

If there was a fire and you survived your insurance may have a few things to say about what you are doing here. The hydrogen and the smell from the batteries can't be good.
 
Last edited:
Skillz said:
More info on the batteries you're using? What charger are you using to charge them? How do you have them connected exactly? Basically like a off the shelf UPS. The equipment is connected directly to the batteries, but power is fed from the wall, in the event of a power failure the batteries take on the load? What equipment are you using to accomplish this?
Click to expand...

These are the batteries:
http://www.canadiantire.ca/AST/brow...Starting+and+Deep+Cycle+Battery.jsp?locale=en

The backup system is a Trip-Lite inverter-charger:
http://www.tripplite.com/en/products/model.cfm?txtSeriesID=818&txtModelID=2938

Basically the system runs on AC all the time, the batteries are standby. When the power goes out it switches to inverter. I'd like to do a telco-like setup with rectifier/inverter but then I'd have to look at the more expensive cells designed to be charged all the time and hydrogen would become a slight concern. Batteries do not produce hydrogen unless they are being overcharged though. When they produce hydrogen they're actually slowly dying as they are losing electrolyte. I'll have a hydrogen sensor for good measure in there though. Probaly add an exhaust with brush-less motor (centrifugal fan should work I think). Could set it on a relay where if the hydrogen sensor kicks in, the fan kicks in.

Only going to add 2 more. If ever I expand and use more I can always look at gel cells though but they're like 3x the price but it would give me better peace of mind.

I would imagine people who have solar power have racks much larger than mine with much larger batteries so if done well it's not a big deal. My hot water heater is a much more dangerous system and has the ability to completely blow up my house if something goes wrong with it.
 
Last edited:
Red Squirrel said:
These are the batteries:
http://www.canadiantire.ca/AST/brow...Starting+and+Deep+Cycle+Battery.jsp?locale=en

The backup system is a Trip-Lite inverter-charger:
http://www.tripplite.com/en/products/model.cfm?txtSeriesID=818&txtModelID=2938

Basically the system runs on AC all the time, the batteries are standby. When the power goes out it switches to inverter. I'd like to do a telco-like setup with rectifier/inverter but then I'd have to look at the more expensive cells designed to be charged all the time and hydrogen would become a slight concern. Batteries do not produce hydrogen unless they are being overcharged though. When they produce hydrogen they're actually slowly dying as they are losing electrolyte. I'll have a hydrogen sensor for good measure in there though. Probaly add an exhaust with brush-less motor (centrifugal fan should work I think). Could set it on a relay where if the hydrogen sensor kicks in, the fan kicks in.

Only going to add 2 more. If ever I expand and use more I can always look at gel cells though but they're like 3x the price but it would give me better peace of mind.

I would imagine people who have solar power have racks much larger than mine with much larger batteries so if done well it's not a big deal. My hot water heater is a much more dangerous system and has the ability to completely blow up my house if something goes wrong with it.
Click to expand...



only if you let the mythbusters install it for you... :D:cool:

modern day water heaters are designed to blow the emergency pressure valve at such safe pressures it's nearly a non issue. only when you're Adam Savage and remove the valve does it become a problem.
 
Very few residential solar systems include batteries. Mostly because they are dangerous. The ones that do require them to be cabinetized outside the footprint of the dwelling.
 
I talked to a co-worker that works in power for the largest telecommunications company in the US about your setup, and his comment was "that guy must want to die". He proceeded to explain that there are codes in place for the use of wet cell batteries indoors for a reason. A even a 4 car size battery setup in a 20'x20'x10' room with >10CFM of circulation can generate enough hydrogen to cause a problem.

Take heed, he has been doing this for 30+ years and maintains about 400 banks of batteries ranging from 100Ah to 30,000Ah.
 
What about if I use catalytic caps? that will basically force the hydrogen to turn back to water. Will essentially convert it into a VRLA, but at a fraction of the price. Just not sure where I'd get such caps.

What I'm wondering is, where exactly does all this hydrogen come from? There is a finite amount of electrolyte/water in the battery and if it was truly producing hydrogen at a rate faster than the rate it is depleting in the atmosphere, wouldn’t the battery completely empty itself out? Ex: if I take a container of water, put a bunch of plates in it, apply power, and let it bubble out till the water is all gone there will still not be all that much hydrogen in the room because it will have mixed with the air and escape through walls, etc. So what makes a battery different?

What if I install a small venmar unit in the server room? Maybe something with two 5 inch pipes and a fan that occasionally runs. Basically needs to be a HRV system otherwise the server room will get too cold if I just pump outside air directly in. If I vent the cabinet out, some air needs to replace it...
 
Last edited:
If you are considering all this work for these batteries you'd be better off going with some APC units and a generator.
 
Red Squirrel said:
What about if I use catalytic caps? that will basically force the hydrogen to turn back to water. Will essentially convert it into a VRLA, but at a fraction of the price. Just not sure where I'd get such caps.

What I'm wondering is, where exactly does all this hydrogen come from? There is a finite amount of electrolyte/water in the battery and if it was truly producing hydrogen at a rate faster than the rate it is depleting in the atmosphere, wouldn’t the battery completely empty itself out? Ex: if I take a container of water, put a bunch of plates in it, apply power, and let it bubble out till the water is all gone there will still not be all that much hydrogen in the room because it will have mixed with the air and escape through walls, etc. So what makes a battery different?

What if I install a small venmar unit in the server room? Maybe something with two 5 inch pipes and a fan that occasionally runs. Basically needs to be a HRV system otherwise the server room will get too cold if I just pump outside air directly in. If I vent the cabinet out, some air needs to replace it...
Click to expand...

Outside air is the safest, but it can be done with inside air. The challenge is creating a sealed environment, which in a house, isn't going to happen. No matter what you will have air leakage above the equipment.

At this point you are going to argue a negative pressure system, which can be done correctly, but again, in a house it is going to be next to impossible to control the air actually being sucked out of the server room.

Not to mention this creates more of a load on the batteries when they are in use. Batteries vent hydrogen when (over)charging and discharging.

Sealed batteries really are the best option for a home. Sealed batteries don't cause homeowners insurance issues, violate building codes and are fairly inexpensive. Get multiple 6V batteries from ebay and wire them up in 2 banks using some buss bars.
 
The inexpensive part is where things change. They're way more expensive. Ebay is not even an option when considering heavy things like batteries, the shipping would be astronomical. Though there is one place in town that sells small sealed batteries for cheap, I'd have to check with them to see what they got as far as bigger ones. If I can get a 100AH battery for say, 200 bucks, might be worthwhile to start with 4 of those and at least I don't have to worry about insurance stuff and all that.

Currently working on testing my hydrogen sensor that I have been wanting to install, going to install that as well on the hot isle, where any hydrogen would be exhausted if I keep my current setup.

Was also looking at calculations here:

http://www.theresourcestore.ca/toolbox-batteries.php

I need maybe 0.074 air changes per hour according to that. I only took rough measurements of the total air cubic meter though.
 
Just use the sealed, UL listed batteries. Frys stocks them...and eBay is more of an option than you think. Shipping is not really prohibitive. Add in the money you'll save on the H sensor, ventalation, peace-of-mind, etc, and you'll find they are not nearly as expensive as you think.
 
Remember, I'm in Canada. Everything is more expensive here. Then there's customs etc... ordering batteries that weigh 60+ lbs is not an option.

I will check to see what the electrical shop here has as far as>=100AH SLA/VRLA batteries, but when I checked Canadian Tire they only have a few and they're in the $300 range vs $100 for equivalent 100AH battery. The ones you see on Ebay are very small batteries for alarm systems and stuff. All way under 100AH.
 
Testing H2 sensor. :D



Don't mind the mess, was practising pipe sweating since I never did it before, all sorts of random fittings lying around. :D

The value goes from 0-1024 and somehow translates to PPM, not too sure how to figure it out but really what matters is that the number increases = more hydrogen. So just need to figure out what is the number where it can ignite at, hence the lighter. That will be "Critical" alarm. Then I can just have major and minor be the lower values. There's also a sensitivity I can set on the sensor. I brought it right down for proper testing. 60-70 seems to be normal air, it goes up to around 100 when I run the electrolysis. Got it up to 200. I may need a better and more sealed setup for testing though as I can't get it to ignite.

High school science class all over again. "Let's blow something up" was pretty much how class started.
 
Last edited:
Well that's a pretty neat setup.

You will have to use something to calibrate it though. Know anyone with a real meter?

"normal" air is 0.5PPM

EPA says 100PPM is a evacuation alarm, explosion imminate

Don't forget that hydrogen displaces oxygen also
 
Last edited:
Yeah I'll have to see if I can find one I work at a telco CO so I would imagine one of the techs might actually have one.

My sensor needs to be on for at least 24 hours to "burn in" though so my current tests arn't exactly accurate right now.
 
Red Squirrel said:
Yeah I'll have to see if I can find one I work at a telco CO so I would imagine one of the techs might actually have one.

My sensor needs to be on for at least 24 hours to "burn in" though so my current tests arn't exactly accurate right now.
Click to expand...

The power tech should, here is the US it's pretty standard. I know the CO power techs here have them to verify that the hydrogen detectors are working correctly.
 
Monitoring system in action. This is just in testing phase still, the sensor is actually in my office connected through a cat6 cable through my patch panel to the monitoring device.




It sends emails too, which I want to also set a pop account for my phone so my phone gets some of the more critical alarms.

Mon: srvrm_h2
Host: hal9000
Value: 101 /1024
Alarm: Server Room Hydrogen Level elevated
State: Major
Date: Feb-13-2013 04:53:08am

Sent by EnviroState 1.0.7
Click to expand...

Going to be adding water sensors around my basement, and I also want to try finding smoke detectors that have alarm point terminals, and monitor that too.
 
I forgot I made this thread, suppose it's time for an update.

Did not start on stuff as soon as I hoped but I did start on the battery rack.



The designed changed a few times as I was building it so it ended up a little over complicated for nothing as I wanted to make due with the parts I already assembled, but it will be nice and solid. Could probably have gotten away without that middle post though, I think I may be underestimating the strength of 3/4" plywood and 2x4's. Then again can never be too safe when it has to hold lead acid batteries.

I still have to cut the side panels, top panel and I have to decide what to do for the front/back. I may just do a door on hinges with a latch. I will prime and paint before I do the side panels though as it will be easier to get everywhere. Probably going to paint it all black. As part of the same project I'll also be making some cable management "ducts" for my server rack using 2x6's and strips of wood to act as a kind of finger duct system. It will also have outlet banks to plug everything. So all of that will be black too, to match the rack. I have another rack that's already fully enclosed so it will stay that way and sit beside. Though, it's beige, so debating on painting it too.

Been doing lot of work on the basement, so now I'm finally at the fun part. :D Can't wait to finish and get all this stuff in order though, it's a mess down there.
 
Red Squirrel said:
I forgot I made this thread, suppose it's time for an update.

Did not start on stuff as soon as I hoped but I did start on the battery rack.

The designed changed a few times as I was building it so it ended up a little over complicated for nothing as I wanted to make due with the parts I already assembled, but it will be nice and solid. Could probably have gotten away without that middle post though, I think I may be underestimating the strength of 3/4" plywood and 2x4's. Then again can never be too safe when it has to hold lead acid batteries.

I still have to cut the side panels, top panel and I have to decide what to do for the front/back. I may just do a door on hinges with a latch. I will prime and paint before I do the side panels though as it will be easier to get everywhere. Probably going to paint it all black. As part of the same project I'll also be making some cable management "ducts" for my server rack using 2x6's and strips of wood to act as a kind of finger duct system. It will also have outlet banks to plug everything. So all of that will be black too, to match the rack. I have another rack that's already fully enclosed so it will stay that way and sit beside. Though, it's beige, so debating on painting it too.

Been doing lot of work on the basement, so now I'm finally at the fun part. :D Can't wait to finish and get all this stuff in order though, it's a mess down there.
Click to expand...

You over-engineered the hell out of that rack. I like it. If you want strong shelves, go with advantech flooring. Costs a lot more, but i've used that stuff to span 8' gaps before (covering the hole for the stairs when building a house), with me working on top of it.

You should stain it black instead of painting it. Looks much better.
 
Hmm never thought of stain, will it give me a very uniform dark black though without seeing the wood grain? I want a flat/slightly reflective black. Basically the same black as metal or plastic.

Also debating on if I want to put the batteries in containers or not. Will be simpler to just slide them right on the shelf, but containers will provide extra protection in case of leak. Issue is trying to find containers that will fit exactly. 4 batteries is pretty much going to be end to end. Might just put some vapor barrier plastic. Batteries are put in cars, boats, RVs and are bounced around. They're solid. It's mostly to deal with the few droplets that might leak when I do maintenance. I think I'll have to take them out to do maintenance anyway, will be too hard to view the levels correctly for the back ones.
 
Red Squirrel said:
Hmm never thought of stain, will it give me a very uniform dark black though without seeing the wood grain? I want a flat/slightly reflective black. Basically the same black as metal or plastic.
Click to expand...

Plasti-dip then, lol.

It's hard to say if you would get a uniform color in a 2x4, they aren't really cut to show off their grain. Its also hard to get a glossy color on low grade wood because it sucks up so much paint. Maybe use a really thick latex or oil based paint. I'm kind of curious what plasti-dip would look like on it though. You can get it in aerosol cans, and it goes on pretty thick. EDIT: or even better, truck bed-liner. The perfect coating for an over-engineered rack.
 
Oh, and on a slight different subject, Some of the server related changes since I last posted:


New file server. Goal is to move ALL files to this, and use NFS/Kerberos/LDAP.


The insides, 3 IBM cards and a fibre channel card for the IBM San enclosures


Racked up!


New firewall installed, core2duo to replace the P3 I had.


2 more batteries to add more capacity to the UPS to compensate for the added load. The main reason for the rack. 2 batteries on the floor is one thing, but when I start to have 4 and possibly more in the future, time for a more dense setup.

Also going to try to get some 2 awg cabling and the proper crimper/ends and run exact lenghts I need going from battery rack to the server rack where I will put the UPS, and in the future, rectifier/inverter.
 
wizdum said:
Plasti-dip then, lol.

It's hard to say if you would get a uniform color in a 2x4, they aren't really cut to show off their grain. Its also hard to get a glossy color on low grade wood because it sucks up so much paint. Maybe use a really thick latex or oil based paint. I'm kind of curious what plasti-dip would look like on it though. You can get it in aerosol cans, and it goes on pretty thick. EDIT: or even better, truck bed-liner. The perfect coating for an over-engineered rack.
Click to expand...

Sounds interesting, though think I'll stick with primer + paint, probably cheaper. Not too too fussy about the look so if I don't get the glossy look I want as long as it turns out half decent. Also been thinking, for the server rack/cable management I may go with a couple PDUs instead of a custom outlet install. If I figure how much my time is worth the PDUs pay for themselves. A couple screws and done, 24 outlets. :p Couple more screws, 24 more outlets.
 
Red Squirrel said:
Sounds interesting, though think I'll stick with primer + paint, probably cheaper. Not too too fussy about the look so if I don't get the glossy look I want as long as it turns out half decent. Also been thinking, for the server rack/cable management I may go with a couple PDUs instead of a custom outlet install. If I figure how much my time is worth the PDUs pay for themselves. A couple screws and done, 24 outlets. :p Couple more screws, 24 more outlets.
Click to expand...

I've got a PDU i've been trying to sell.....its a 208v version though.
 
Yeah all my stuff is 120 single phase. I was looking at Tigerdirect real quick and I can actually get a 16 outlet metered Tripp Lite for a bit over 100 bucks. I like the idea that lot of them are metered too. I'd probably get like 4. 2 on each side, then I can plug in different UPSes/sources. One plugged into surge only, maybe two in my big UPS and one into the smaller UPS. May as well take advantage of stuff that has redundant PSU and split it off to different power sources.
 
I have a couple basic APC 1U Basic 15A 110V PDUs if you are interested.
 
Might take you up on that offer, but I'll hold off for now. How much would you be asking per unit and do you have a model number or pic?
 
You must log in or register to reply here.
Back
Top