![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
A small datacenter build and other projects
- Thread starter The Spyder
- Start date
jeffmoss26
2[H]4U
- Joined
- Aug 1, 2002
- Messages
- 2,267
Nice stuff as always! I think I have the same db meter, I got it when I was running sound. VERY helpful!!
I use free-pbx works great. Not sure what spyder is using tho..
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
I downloaded and installed FreePBX for right now. This is just for eval purposes. I have a few local friends who build/sell/support Asterisk based systems that recommended it. One is running 50 Yealinks/FreePBX and with the recommendations I saw here and from other sites, I wanted to give them a try. Nice phones so far. This is my first real dive in to phone systems outside of old PBX's, so there is a bit of a learning curve.
MikeTrike
[H]F Junkie
- Joined
- Nov 16, 2005
- Messages
- 12,169
Me! We've got 142~ extensions running on it. A mix of Polycom 650's primarily. With some Yealinks peppered into the mix. As well as some Grandstream ATA's for analog stuff like paging and cordless phones.
We've been running it for about a year now. Our only current hiccup is some SIP issues with nexVortex and an issue with 3CX v11 that does not show when you have a missed call from an outside line on our Polycoms. However we have a support ticket open with 3CX on that one and will be moving our SIP services away from nexVortex. Possibly to Time Warner in the next 9-12 months. Maybe sooner depending on what we have going on.
Red Squirrel
[H]F Junkie
- Joined
- Nov 29, 2009
- Messages
- 9,211
More awesome stuff Spyder!
Also a side story about DB Meters, since your post reminded me of it and thought it would be funny to share. So I work in a telco central office and we had a coop student. It's a tradition to eventually get the student to go get a bucket of dial tone. It's the telco version of sky hooks. Well we did something different. Gave the student a db meter to go check the distribution frame for dial tone leaks.
Also a side story about DB Meters, since your post reminded me of it and thought it would be funny to share. So I work in a telco central office and we had a coop student. It's a tradition to eventually get the student to go get a bucket of dial tone. It's the telco version of sky hooks. Well we did something different. Gave the student a db meter to go check the distribution frame for dial tone leaks.
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
Well, it finally happened- after 9 months I have burnt out. Last week was just hellish. I got incredibly frustrated over a simple fix, and I hardly ever get truly ticked. We have had some very very stressful projects going through the production side of things and the stress is getting to everyone. My solution: re-group, get through 200 resumes, and get someone hired. It's going to end up being a slow week, but otherwise I would go insane
More pictures of what little I got done last week.
I asked our vendor if they had one of the new i7 ITX supermicro boards, this showed up.
And this lovely little D525 atom came with it. (2x1.8ghz, 4gb, 40gb SSD- takes a DC power brick input.)
Some very, very heavy boxes showed up.
And so did nearly 700tb of disks + controllers.
Got all the racks blocked off.
I keep telling my boss it would be cheaper just to fill them full of servers, but for some reason he thinks I'm full of it. haha.
Found this today after some office shuffling- hope the monitors OK- they are temperamental suckers.
More pictures of what little I got done last week.
I asked our vendor if they had one of the new i7 ITX supermicro boards, this showed up.
And this lovely little D525 atom came with it. (2x1.8ghz, 4gb, 40gb SSD- takes a DC power brick input.)
Some very, very heavy boxes showed up.
And so did nearly 700tb of disks + controllers.
Got all the racks blocked off.
I keep telling my boss it would be cheaper just to fill them full of servers, but for some reason he thinks I'm full of it. haha.
Found this today after some office shuffling- hope the monitors OK- they are temperamental suckers.
Last edited:
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
Those are Supermicro 2u 24bay 900w.
http://www.supermicro.com/products/chassis/2U/216/SC216A-R900LP.cfm
http://www.supermicro.com/products/chassis/2U/216/SC216A-R900LP.cfm
Spyder, be careful. I have personal experience with a health crisis that was a result of running myself @ redline for months... Not pretty.
Get that IT assistant hired, so we can continue to enjoy and be inspired by the good work.
Hi again,
I notice that you use mostly Supermicro server barebones, especially for storage. I want to use them for my Medical IT business thats about to start-up in 2013. Any pros and cons that you would like to share about your experience with their products? How are they with warranty claims etc.
Last edited:
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
I caught myself well before I had any physical issues. It happens to everyone here at some point, we all work ourselves down, take a bit to recover, and go back at it. I have sadly watched it happen a few times to coworkers.
As far as the Supermicro boxes, we deal mainly with IXsystems who handles the assembly, burn in, and RMA's. For internal use, I build my own using CDW/Provantage/Newegg. No complaints so far. IMO, they are great boxes. We have over 1000 deployed with only a handful of bad powersupplies being the
biggest issue. (High heat/Desert environment)
As far as the Supermicro boxes, we deal mainly with IXsystems who handles the assembly, burn in, and RMA's. For internal use, I build my own using CDW/Provantage/Newegg. No complaints so far. IMO, they are great boxes. We have over 1000 deployed with only a handful of bad powersupplies being the
biggest issue. (High heat/Desert environment)
Red Squirrel
[H]F Junkie
- Joined
- Nov 29, 2009
- Messages
- 9,211
Nice to see those Supermicros are treating you well, I'm strongly considering going that route from now on for my server stuff. Been thinking about upgrading my whole environment. Wont be a while till I do that though, have to save up first. Those 2.5" enclosures look pretty nice... I never really considered 2.5" drives for mass storage but considering even at the consumer level you can get like 1TB 2.5" drives now it's definitely a neat option. Think they are less susceptible to vibration issues too.
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
Sadly I am working on quite a bit of production tasks (Writing guides/researching tape backups for 5.xPB of data...) and I have very little to post.
Here is a small teaser of other projects I have been working on- Mike will recognize this.
Here is a small teaser of other projects I have been working on- Mike will recognize this.
Red Squirrel
[H]F Junkie
- Joined
- Nov 29, 2009
- Messages
- 9,211
Almost looks like industrial size electrical tape. What someone would use if they nicked a 500,000 volt cable by accident. "Bah, just wrap it in electrical tape and call it a day"
Says PVC Strip though, I'm guessing it's something else, some kind of covering for something.
Says PVC Strip though, I'm guessing it's something else, some kind of covering for something.
klank
Killer of Killer NIC Threadz
- Joined
- Aug 22, 2011
- Messages
- 2,177
My guess is its for a hot isle air curtain.
However it does look like a massive roll of electrical tape.
However it does look like a massive roll of electrical tape.
yes it does took like tape,
but as stated before it must be for a curtain to separate the hot/cold air.
But how wide is it , because it looks to narrow to me to be the actual curtain
but please keep on posting the pics spyder, its really cool to see that stuff you do.
but as stated before it must be for a curtain to separate the hot/cold air.
But how wide is it , because it looks to narrow to me to be the actual curtain
but please keep on posting the pics spyder, its really cool to see that stuff you do.
Last edited:
MikeTrike
[H]F Junkie
- Joined
- Nov 16, 2005
- Messages
- 12,169
You'd be correct in assuming it was for hot/cold isle separation. As those vents on the floor are the cold air and the vents on the ceiling behind the cabs looks like the return air. No sense in mixing the air up, I did the same thing with our two cabs. We'll be doing the same thing when we build our new building as well.
I would not be surprised if they used blocking plates in the cabs with empty space as well.
Still on a smaller scale though.
Either that or it's late and I'm mixing up comments. Either way I'll find out tomorrow.
I would not be surprised if they used blocking plates in the cabs with empty space as well.
Still on a smaller scale though.
Either that or it's late and I'm mixing up comments. Either way I'll find out tomorrow.
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
Mike is correct-
It's 8" wide Anti-Static curtain material for hot/cold isle separation.
I have a roll of 48" on order too.
336TB and two controllers magically appeared in my rack.
Next to it is 144TB- same project. Almost have a Petabyte racked now. Can't wait to get my other 336TB SAN. Wayyyy to much storage
Oh and racecar runs again
(It got a fresh motor/turbos/larger injectors/better clutch/tons of freaking time and money.... They don't call them (F)inancial (D)isasters for nothing)
It's 8" wide Anti-Static curtain material for hot/cold isle separation.
I have a roll of 48" on order too
. 336TB and two controllers magically appeared in my rack.
Next to it is 144TB- same project. Almost have a Petabyte racked now. Can't wait to get my other 336TB SAN
. Wayyyy to much storage Oh and racecar runs again
(It got a fresh motor/turbos/larger injectors/better clutch/tons of freaking time and money.... They don't call them (F)inancial (D)isasters for nothing
)The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
Magical spinning triangles on a stick?
Boost in-> Apex seals out?
3 moving parts-> 99 possible problems??
I can make fun of myself right? Honestly, there is not another car that drives like a 3rd Gen RX7 for under $50k.
Boost in-> Apex seals out?
3 moving parts-> 99 possible problems??
I can make fun of myself right? Honestly, there is not another car that drives like a 3rd Gen RX7 for under $50k.
wra18th
[H]F Junkie
- Joined
- Nov 11, 2009
- Messages
- 8,365
Good to see another Mazda guy. I have two. 2003 6S and 2009 CX-9 Grand Touring. Yeah, the rotarys are weird. Saw a four rotor on youtube yesterday.
Magical spinning triangles on a stick?
Boost in-> Apex seals out?
3 moving parts-> 99 possible problems??
I agree that the car itself was built incredibly well in terms of handling. Damn near 50/50 weight distribution, but they just need to do something with the actual driving force of the car.
they make some cool power, however they make more heat than anything else, 90% of the issues are heat related..
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
Let's not turn this in to a Rotary thread . I do agree the RX8 has a terrible engine/coolant seal issue and don't get me started on idiots at Manna- the reman company. The FD however mainly suffers from a lack of maintenance. I have flogged the every living heck out of mine for years and never had a engine issue, outside of the control system no longer turning on the primary turbo due to a bad check valve.
. I do agree the RX8 has a terrible engine/coolant seal issue and don't get me started on idiots at Manna- the reman company. The FD however mainly suffers from a lack of maintenance. I have flogged the every living heck out of mine for years and never had a engine issue, outside of the control system no longer turning on the primary turbo due to a bad check valve. The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
So a new theoretical was placed before me today: Build a 60 node cluster.
Our current hardware would give us 720 cores/11.5TB ram/15.5TB of localized storage for 30U of rack space and need 2 36 port IB switches, 2x 48 port for IPMI, and 30 power cords.
Switching to 6 7U Twinblade chassis would provide 1920 cores/61TB ram/28.8TB of localized storage/Built in 40GB QDR IB with no cabling, 24 power cords, and 12 network cables. Sadly it costs about $400k more then what the current system would....
I have no idea how a blade system would hold up in the environment we are working with.
224- Cores per 7u 4 processor 8 core per processor
320- Cores per 7u TwinBlade 4 processor 8 core per processor
Our current hardware would give us 720 cores/11.5TB ram/15.5TB of localized storage for 30U of rack space and need 2 36 port IB switches, 2x 48 port for IPMI, and 30 power cords.
Switching to 6 7U Twinblade chassis would provide 1920 cores/61TB ram/28.8TB of localized storage/Built in 40GB QDR IB with no cabling, 24 power cords, and 12 network cables. Sadly it costs about $400k more then what the current system would....
I have no idea how a blade system would hold up in the environment we are working with.
224- Cores per 7u 4 processor 8 core per processor
320- Cores per 7u TwinBlade 4 processor 8 core per processor
If your environment involves high heat and sand like i think you are talking about, it will not be good.
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
I ended up spec'ing the following to replace our 1366 based systems.
1U Supermicro TWIN 2011 E5-2600 with 40GB Infiniband
4x E5-2670 2.66ghz/20MB cache/8 core
16x 16GB DDR3 1600 ECC REG
2x Intel 520 240GB SSD
$10.5k in just parts (fair market value)
Should be getting one ordered here in the next few weeks for testing. This gives us 256GB of ram/32 Cores per 1U vs the existing 196GB/24 Core we currently run. Less power too!
1U Supermicro TWIN 2011 E5-2600 with 40GB Infiniband
4x E5-2670 2.66ghz/20MB cache/8 core
16x 16GB DDR3 1600 ECC REG
2x Intel 520 240GB SSD
$10.5k in just parts (fair market value)
Should be getting one ordered here in the next few weeks for testing. This gives us 256GB of ram/32 Cores per 1U vs the existing 196GB/24 Core we currently run. Less power too!
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
Repeat "I love my job." Even at a cool place like this, politics can suck, burn out happens, and time gets wasted. My only solace is for every 1 BS thing I have to deal with, I get to do 10 cool things . Hiring has been temporarily frozen, so has capital expenditures. Combine those with long hours on 30 projects at a time = burned out. It's been 2-3 weeks now and I haven't even began to recover. Hopefully this is just a small bump in the road. I have a failing SAN that holds our main company files, no backups for the VM hosts, and a ton of infrastructure spending coming up next year. The biggest being AC for the datacenter. Most importantly though is I need a helper, like yesterday. Rumor has it this is only until the 1st week in December after a visit from the main program managers is over.
/Rant.
I have some really neat photos to show you guys once I find my card reader.
. Hiring has been temporarily frozen, so has capital expenditures. Combine those with long hours on 30 projects at a time = burned out. It's been 2-3 weeks now and I haven't even began to recover. Hopefully this is just a small bump in the road. I have a failing SAN that holds our main company files, no backups for the VM hosts, and a ton of infrastructure spending coming up next year. The biggest being AC for the datacenter. Most importantly though is I need a helper, like yesterday. Rumor has it this is only until the 1st week in December after a visit from the main program managers is over./Rant.
I have some really neat photos to show you guys once I find my card reader.
Repeat "I love my job." Even at a cool place like this, politics can suck, burn out happens, and time gets wasted. My only solace is for every 1 BS thing I have to deal with, I get to do 10 cool things
/Rant.
I have some really neat photos to show you guys once I find my card reader.
Awesome, can't wait to see them
Repeat "I love my job." Even at a cool place like this, politics can suck, burn out happens, and time gets wasted. My only solace is for every 1 BS thing I have to deal with, I get to do 10 cool things
/Rant.
I have some really neat photos to show you guys once I find my card reader.
I thought you had been able to hire an assistant already? Has that still not happened??
heh i would have loved to be a part time student intern/coop at your place... work for free and get job experience and a letter of recommendation... might look into that if that's possible, good for the physical time consuming stuff without letting them be dangerous...
Having the flexibility and resources to set things up as you want is a great gift... I've had to finagle and haggle to set things up the way I want... didn't setup backups because I focused on the steps to recover from failure... and we had no budget for it...
They can only push you as far as you let them... Remember to stop and breathe... take a 3 day weekend at least once a month... call out sick and let them consider what happens if you really get sick...
Having the flexibility and resources to set things up as you want is a great gift... I've had to finagle and haggle to set things up the way I want... didn't setup backups because I focused on the steps to recover from failure... and we had no budget for it...
They can only push you as far as you let them... Remember to stop and breathe... take a 3 day weekend at least once a month... call out sick and let them consider what happens if you really get sick...
The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
So as it turns out, everything worked out. A Teir 1 tech was brought on for another project that got canned, so I adopted him. He doesn't start for two weeks, but it's better then nothing! This week several program managers from the government department that oversees our programs are visiting for a inspection. In prep we reorganized our warehouse, cleaned, and I got a ton done.
Anyways, pictures!
Did I mention I refuse to step foot in the R+D lab? Jebus. Tie down straps on $30k of SAN on 3" of foam. Can''t blame them though, lost 9 disks to the sound system from the bar below. I just lost 6 in my in house storage for the same reason. My solution is to move the SAN to our warehouse racks though.
Found this in there too. L6-30P to two L6-30R's.
Added it to the collection of other WTF were you thinking electrical nightmares.
My curtain showed up.
In the mean time, I built a small workstation for system testing/validation/troubleshooting. Missing is the antistatic mats.
Aluminum showed up for the wall.
Finally the hangers showed.
Starting on the wall... the building is 1/4 out of square over 20ft... sweet!
Built a TV mount for the small conference room, looks 10x better then a folding table.
Uh-oh. Someone opened the mains... And I worked 12h on Saturday.
For my new 200amp 240v 3ph panel! I can finally use all 400 amps of service instead of the 350 the last panels limited me to.
Added some more outlets for the racks.
And another 200amp 3ph panel in prep for a 400amp 480v service and AC.
My boss came in over the weekend and pressure washed the driveway for our visit from the government project managers...
Racks have not changed much.
Added another full depth rack to move the above mentioned SAN from our headquarters into.
More wall progress.
And it is done!
So far we have a 20db drop from 20ft away, which is huge. It means I will need AC ASAP, but I should make it in to the spring no problem. The wall structure is 2x2x1/4 6063 aluminum and should easily support a matching roof system to finish off the cold isle. AC will be a ducted in from rooftop units. Did I mention both my boss and the CEO now think I am some crazy fabricator/car guy/IT guy.... Wait a minute! They're right!
The next update will probably be a bit. I am taking a long needed vacation and working on rather boring projects. Hopefully I will start ordering hardware to rebuild my VM cluster and triple our in house cluster. Until then! Servus!
A Teir 1 tech was brought on for another project that got canned, so I adopted him. He doesn't start for two weeks, but it's better then nothing! This week several program managers from the government department that oversees our programs are visiting for a inspection. In prep we reorganized our warehouse, cleaned, and I got a ton done.Anyways, pictures!
Did I mention I refuse to step foot in the R+D lab? Jebus. Tie down straps on $30k of SAN on 3" of foam. Can''t blame them though, lost 9 disks to the sound system from the bar below. I just lost 6 in my in house storage for the same reason. My solution is to move the SAN to our warehouse racks though.
Found this in there too. L6-30P to two L6-30R's.
Added it to the collection of other WTF were you thinking electrical nightmares.
My curtain showed up.
In the mean time, I built a small workstation for system testing/validation/troubleshooting. Missing is the antistatic mats.
Aluminum showed up for the wall.
Finally the hangers showed.
Starting on the wall... the building is 1/4 out of square over 20ft... sweet!
Built a TV mount for the small conference room, looks 10x better then a folding table.
Uh-oh. Someone opened the mains... And I worked 12h on Saturday.
For my new 200amp 240v 3ph panel! I can finally use all 400 amps of service instead of the 350 the last panels limited me to.
Added some more outlets for the racks.
And another 200amp 3ph panel
in prep for a 400amp 480v service and AC.
My boss came in over the weekend and pressure washed the driveway for our visit from the government project managers...
Racks have not changed much.
Added another full depth rack to move the above mentioned SAN from our headquarters into.
More wall progress.
And it is done!
So far we have a 20db drop from 20ft away, which is huge. It means I will need AC ASAP, but I should make it in to the spring no problem. The wall structure is 2x2x1/4 6063 aluminum and should easily support a matching roof system to finish off the cold isle. AC will be a ducted in from rooftop units. Did I mention both my boss and the CEO now think I am some crazy fabricator/car guy/IT guy.... Wait a minute! They're right!
The next update will probably be a bit. I am taking a long needed vacation and working on rather boring projects. Hopefully I will start ordering hardware to rebuild my VM cluster and triple our in house cluster. Until then! Servus!