Project Log: - Office Move with Pics

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Background:

The lease is up on our building in Burlingame, CA so my company decided to look around for better digs to call home. After looking around and exploring various options, we found a spot in downtown San Francisco. The unfortunate thing was that to build a server room into our suite was going to cost us around $200,000, thanks to the building and San Francisco requiring union labor and requiring us to pick form a very, very short list of contractors (read: 1 company) who have a monopoly on any work done in the building.

Instead we decided to host 99% of our gear in 365 Main a colo approximately two blocks away from our suite.

The final result is that as far as IT is concerned, I'll be managing two moves starting next month: one into the colo space and one smaller move into our office.

What follows is a work log of my progress, with pictures as I can manage to take them.



In the simplest form, here's our network layout:

networklinks.png


Our office will access resources at the colo and use the uplink at the colo for internet access. This removes the need for any firewall equipment in our office. VPN traffic from remote users will be straight to recourses in the colo.



The first part of this project has been negotiations with Cogent and 365 Main and signing contracts. All boring as hell. But now that those are signed, we can move stuff in:

IMG_0044.jpg


I had to do a lot of port shuffling to free up these switches from our existing stacks. I also had to break our HA firewall pair to use one in the colo. When all is said and done and we've moved into our new office, the second firewall will be pulled from the existing office and mated with this guy at the colo.



It all starts here:

IMG_0053.JPG


There's nothing in the five cabinets except for a pair of fiber cross-connects. One L3 100mbit link to the internet, one gigE L2 link to our new office. These went in yesterday and now I'm waiting for Cogent to hook up their end. When that goes live, I can bring in the routers and start racking gear.
 
Last edited:

ciggwin

Supreme [H]ardness
Joined
May 30, 2006
Messages
4,861
I'm starting to plan an office move as well - I'll be following this thread closely! Keep the pics comin :p
 

Adam

[H]ard|Gawd
Joined
Jan 9, 2003
Messages
1,592
Ohhh i don't envy you :) We just moved ourselves (another topic) and we're still finishing things up as far as our server room goes. Gluck is all im gonna say, lol.
 

RiDDLeRThC

2[H]4U
Joined
Jun 13, 2002
Messages
3,961
Interesting, I would hate to have to trust all our internal traffic to an outside connection but if the SLA's are in place I guess its not a big deal. Not too sure why I worry so much. In the two years we have been in our new building we have had two power outages that resulted in equipment having to shutdown and our T1 drop. Our fiber line on the other hand has been rock solid.
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Interesting, I would hate to have to trust all our internal traffic to an outside connection but if the SLA's are in place I guess its not a big deal. Not too sure why I worry so much. In the two years we have been in our new building we have had two power outages that resulted in equipment having to shutdown and our T1 drop. Our fiber line on the other hand has been rock solid.

This was a big hesitation on our part as well. A cut fiber between the two sites and we might as well send everyone home to work via VPN.

The risk here is mitigated by the fact that this is 'only' our corporate and development network, so a cut fiber won't bring down our production web site, which is hosted on the East Coast.
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
You could always set up a publicly-routed tunnel as a backup route

True, but then we'd need a second line at our corporate office that has access to the internet, plus firewalls, etc. Which we don't want to pay for. Right now the existing link is a layer 2 point-to-point between the two sites only.

My company is pretty cheap and figures that they'll deal with it when it happens. :shrug:
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Two domain controllers and a utility server arrived today

HP DL120 servers with a single quad-core CPU, 4GB RAM, 2x 160GB SATA in RAID-1

IMG_0059.JPG




...aaand some Panduit cable management (front only, 1U and 2U) since we aren't doing structured wiring between cabinets. One cable all the way from server to switch.

IMG_0060.JPG




I promise to take pictures with something other than my iPhone when I get something significat to take pictures of. :D
 
Last edited:

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Plenty of work done and pics are inbound, but for now I'll say that the 100mbit internet link is up and router is configured, firewall is up with a basic config and switches are VLAN'd and sliced and diced.

In the meantime, here's a speed test:

 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Here's pics from the first day of moving in.

Cabinets!

00%20empty-cabinets-front.jpg




Before we moved in, I had the rack monkeys install some vertical cable management:

00%20empty-cabinets-back.jpg




Cart of parts:

01.JPG




Servers and console boxes:

02.JPG




PDUs in place:

03.JPG




Network gear and cable management:

04.JPG




Routers, firewall and switches oh my!

05.JPG




Side shot of network gear (uncabled):

06.JPG
 
Last edited:

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
More work since the last time:




Network core - racked and getting cabled
IMG_1992.JPG


Network core from the rear - no cable management yet
IMG_1992d.JPG


2DC, KVM & Console, utility server racked
IMG_1995.JPG


KVM Console Open
IMG_1997.JPG


Windows 2008 R2 Standard getting installed
IMG_2000.JPG


Front of KVM and DCs
IMG_2001.JPG


Cabling the routers, public scwitch and firewall
IMG_2005.JPG


Servers cabled to switches
IMG_2008.JPG


Cable management
IMG_2009.JPG


Back of utility server after cabling
IMG_2010.JPG


Back of KVM and DCs after cabling
IMG_2012.JPG


view of network cabinet from outside of cage
IMG_2017.JPG


Artsy shot of cage through a neighbor's cage
IMG_2017a.JPG
 
Last edited:

calvinj

[H]ard|Gawd
Joined
Mar 2, 2009
Messages
1,738
what are the 1u dell boxes that are 1u above the 48 port switches. Sorry If you already said what they were
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Those are additional power suuplies to provide power over ethernet. While we are only going to have one IP phone in the datacenter, we had the modules from our office installation so I decided to keep them all together.
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Pretty much. The switch has to be capable of doing PoE, but the extra power supply is what drives it.
 

RiDDLeRThC

2[H]4U
Joined
Jun 13, 2002
Messages
3,961
I was wondering why you had POE power supplies in a DC. lol I should read instead of just looking at pictures.
 

RiDDLeRThC

2[H]4U
Joined
Jun 13, 2002
Messages
3,961
If you dont mind me asking, what ended up being the cost of the fiber link to the DC monthly?
 

Adam

[H]ard|Gawd
Joined
Jan 9, 2003
Messages
1,592
Neat... i still think the best place I ever co-lo'd was a place where i got to touch the dslreports.com server :) Well i didnt actually touch it, but my servers were located near their servers, lol
 

RiDDLeRThC

2[H]4U
Joined
Jun 13, 2002
Messages
3,961
I'm not that trusting of a fiber link lol. I dont think the costs would work for us either. Thats a few racks worth equipment though so I could see it being worth while for you guys maybe depending on the costs of a Small DC buildout in your new building. We only have one rack so building a small room with its own AC wasn't too bad for us.
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Here's how our cost benefit analysis broke down:

The cost of the 100mbit internet link plus the gigE point to point is around $3000/month.
The colo charges another $4000, making the total around $7000/month.

It would have cost us $200,000 for a server room build out.

Our lease is a two year lease, meaning we are going to have to renegotiate everything in two years, which is four months shy of the breakeven point where it is cheaper to build our own server room. The powers that be decided that spending $7k a month for 28 months is better than spending $200,000 up front in a space that we probably won't stay in.

Neat... i still think the best place I ever co-lo'd was a place where i got to touch the dslreports.com server :) Well i didnt actually touch it, but my servers were located near their servers, lol

The speedtest.net server in San Francisco is in 365 Main I think. At any rate, in my other cage here at 365 that I support for another customer, our humble 10mbit link gets some rediculous results when testing against SF: something like 300mbits up and 450mbits down with 0 latency. :D

I'm not that trusting of a fiber link lol. I dont think the costs would work for us either. Thats a few racks worth equipment though so I could see it being worth while for you guys maybe depending on the costs of a Small DC buildout in your new building. We only have one rack so building a small room with its own AC wasn't too bad for us.

We decided that if the link goes down between our office, then everyone gets free "you get to work from starbucks or from home and VPN in" days until it comes back up. Assuming that whatever drops our link at Cogent doesn't affect our metro link as well. If that happens it's going to get really crowded around the KVM in the cage. :D

In our current building, we had literally maxed out the capacity on everything in the server room: When I put the last 2U database server into the absolute last space available in the rack, my UPS on the last available circuit in the panel in our floor hit 80% capacity, driving the battery time down to 5 minutes or less. It also raised the temperature in our server room 1-2degrees meaning that we'd also hit the limit on our AC unit.

So the cost of adding a single 1U server or device after that would have been:

-the cost of knocking down a wall and expanding the server room area plus
-the cost of another 42U cabinet plus
-the cost of adding another electrical panel and 20 amp circuits plus
-the cost of adding another AC unit.

all that to add a single additional Rack Unit. How's that for capacity planning? *flex* Everything topped off at the exact same point.
 
Last edited:

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
I moved into the colo this weekend. I used a company called Silverback to move the gear and it went well.

Still, I had to re-IP everything and reconfigure everything and it's been three really, really long days. I took plenty of pics and I'll be posting them after I get everything back up and operational and then grab some badly needed sleep. :)
 

Sp00n

n00b
Joined
Apr 25, 2006
Messages
39
Nice "little" setup you have here, was just wondering what those cable tidies are that you are using?
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
The horizontal cable management is Panduit. The vertical stuff is some unbranded stuff provided by the Colo facility that has been screwed into the racks themselves with self-tapping screws.
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Here's the first batch of shots from our move over last weekend. This set is basically of Silverback coming in and un-racking the gear we'd identified with pink PostIts.


Shot of the cabinet ready to be gutted:

03.JPG



front of cabinet with pink tags on servers to be pulled

01.JPG



cabinet 1 and telcos 1 and 2

04.JPG



top of telco racks

05.JPG



bottom of telco racks. the dangling cat-5 are from some ShoreTel gear that was pulled for the new office.

06.JPG



pulling cables

07.JPG



pulling servers

07a.JPG



loading the gear onto carts

08.JPG



more gear. another cart.

09.JPG



another another cart

10.JPG



shrink wrapping the cart

11.JPG



wrapping another cart

12.JPG



into the truck with you!

13.JPG



a sad looking cabinet

14.JPG



spaghetti - gotta get the intern on that!

14a.JPG



telco racks post-pull

15.JPG



the remaining user-facing gear goes to our office

16.JPG
 
Last edited:

calvinj

[H]ard|Gawd
Joined
Mar 2, 2009
Messages
1,738
I See SHORETEL!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.

EDIT:

I will miss my shortel call manager and phone in about a week when I change jobs. I really will..
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
Hard to miss those bright orange beacons of VoIP. :)

Our vendor has been really cool and set up an additional MLPS circuit for us so we could run both offices in parallel as we make the transition. We don't have our main data circuit up yet, but by god we have working phones already.
 

calvinj

[H]ard|Gawd
Joined
Mar 2, 2009
Messages
1,738
The new place I'm going too wants to move from their pbx into VOIP. I'm thinking shoretel is where it's at. Funny thing is so do they.
 

xphil3

[H]ard|Gawd
Joined
Nov 11, 2005
Messages
1,212
The new place I'm going too wants to move from their pbx into VOIP. I'm thinking shoretel is where it's at. Funny thing is so do they.
shoretel is where its at?!?!?! I thought that I had you converted with CUCM man...... :eek:

agrikk, very nice stuff as usual. Coming along nicely :D
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
933
ShoreTel gear has been nothing but good to us. I would mos' def' recommend them.
 

Protoform-X

[H]ard|Gawd
Joined
Jan 30, 2002
Messages
1,203
The new place I'm going too wants to move from their pbx into VOIP. I'm thinking shoretel is where it's at. Funny thing is so do they.

Ick. Get out of that comfort zone and grab some real gear. I've replaced several Shoretel systems with Cisco CME/CUE setups. I've gotten plenty of compliments from the users about how much they love their new phones- No one's ever said "I wish I had that Shoretel back!" LOL

That said, I'm not saying Shoretel is bad... It's just more of a Honda Civic whereas the Cisco would be a Lamborghini Murcielago. Plenty reliable, just no class. :)
 

Wayshot

n00b
Joined
Mar 3, 2003
Messages
24
That made me chuckle because I'm an intern now and I'm glad I haven't been given that duty. :D

Be prepared, be prepared! I had an internship at the IT department of the local municipality administration about three years ago. After a server room move, me and two apprentices got the job of cleaning and sorting out the spaghetti mess of network, power and other cables from the old server room.
 

RiDDLeRThC

2[H]4U
Joined
Jun 13, 2002
Messages
3,961
Be prepared, be prepared! I had an internship at the IT department of the local municipality administration about three years ago. After a server room move, me and two apprentices got the job of cleaning and sorting out the spaghetti mess of network, power and other cables from the old server room.

Did they want to reuse the cable? I heard its bad todo that. I never have

The cables get bent in one way for so long when you remove them and start re-bending them different ways you can have issues with the cable breaking or becoming weak inside.

Could be total crap but it does make sense.
 

calvinj

[H]ard|Gawd
Joined
Mar 2, 2009
Messages
1,738
Ick. Get out of that comfort zone and grab some real gear. I've replaced several Shoretel systems with Cisco CME/CUE setups. I've gotten plenty of compliments from the users about how much they love their new phones- No one's ever said "I wish I had that Shoretel back!" LOL

That said, I'm not saying Shoretel is bad... It's just more of a Honda Civic whereas the Cisco would be a Lamborghini Murcielago. Plenty reliable, just no class. :)

Really? Interesting to hear because everyone that I've run into that has started out with Shortel loves it. Doesn't want to go anywhere else. Maybe it's because they don't any different, but I think my biggest thing is how simple it is. I've played with CUCM and it took me over an hour just to figure out where to get a damn phone registered. Xphil had to finally point me in the right direction.

I definitely don't disagree with you. My CME setup at home is great and has alot of cool stuff with it, but I would be torn either way to recommend it. I suppose it would really come down to cost.
 

slayer9019

[H]ard|Gawd
Joined
Apr 2, 2010
Messages
1,599
That made me chuckle because I'm an intern now and I'm glad I haven't been given that duty. :D

God i wish i had an intern!!

Ick. Get out of that comfort zone and grab some real gear. I've replaced several Shoretel systems with Cisco CME/CUE setups. I've gotten plenty of compliments from the users about how much they love their new phones- No one's ever said "I wish I had that Shoretel back!" LOL

That said, I'm not saying Shoretel is bad... It's just more of a Honda Civic whereas the Cisco would be a Lamborghini Murcielago. Plenty reliable, just no class. :)

lol love the quote!!



Move looks like its going good! Last job (got laid off) We had to move a location and merge another location with our main was a PITA. It was me and some other guy (totally worthless) working both the truck and the actual move. Took 2 horribly long weeks dealing with virtualizing while we had to deal with crappy Lightpath. (we paid for a dedicated line and 250 ext. IPs...it took them 5+ months to get the IPs in place!!! WTF??!?!) Colos are the way to go.

Props on the good work.

Also I wouldnt use the old patch cables cause they degrade if you resue them. Just a tip from a past problem I had.
 
Top