multi-year multi-million infrastructure overhaul

C

cyr0n_k0r

Supreme [H]ardness
Joined
Mar 30, 2001
Messages
5,360
Let me start this log by saying that what you will see here is actually about 20 different projects all rolled up into a single multi-year infrastructure overhaul. When I started at this school district there was no money for anything. Technology infrastructure had been neglected for several years as the district had never had an engineer on staff. Below you will find pictures and descriptions of this over-haul now that it has been completed. I have been compiling these pictures over the past several years once I knew the upgrades were going to happen. Some of the pictures will look better than others as several different cameras were used during different phases.
Smaller projects that did not require special certifications (i.e. HVAC, Electrical) I completed in-house with my own staff. Vendors would be called in to take care of special sections such as upgrading an electrical panel, etc. For our very large projects we brought in a VAR that provided us the manpower necessary while I project managed.



The west side of our district office MDF. These are the earliest pictures I have of the room when I began working here. By this time I had already cleaned up a little from what it originally looked like on my first day. There were boxes piled up in front of the door so high that you couldn’t open the door all the way. There were UPS units that were brand new in the box that had been sitting there for 2 years.
From left to right. Our NEC PBX system. There is an even older PBX chassis sitting on the floor that does nothing but condition the power for the NEC MUX. Next is our MUX and power conditioners. Also some marine batteries on the bottom. Then our CSU/DSU’s for the voice T-1’s to each school. Below them are monitors for a security system that no one is sure ever actually worked.
After the walkway gap is our Cisco 4500 and routers. The next 2 racks are all district servers. This is prior to any virtualization. Finally peaking in on the right is our Qwest MUX for the T3 that delivers 2 T1’s for our PRI’s.



The east side of the district office MDF. By this time I had already been able to get some new APC cabinets in to prepare the room for all the projects that were to come. You can also see 2 HVAC mini split units on the south wall. Each one is a Fujitsu 2 ton 19 SEER ductless unit. The room originally had a single 2 ton ceiling unit that was about 5 years old that had never been maintained. It was never able to keep the room under 80 degree even though it ran 24/7. These new units keep the room right at 72 degrees and we run them lead lag. Every quarter I switch them.



The north east corner of the district office MDF. What you are looking at is our Qwest fiber demark for both our Metro Optical WAN, and the T3 for the PRI’s. In the center is a 100 amp single phase panel that provides power for the receptacles in the room. On the right is a 100 amp 3-phase disconnect for the main UPS unit in the room. It is a APC Symetra 8Kva N+1 unit sitting on the floor.



The first phase of cleanup in the district office MDF. You see our Qwest fiber MUX has been removed to reveal the APC Symetra on the floor. We were also able to get rid of some of the 2 post racks as most of the servers were re-racked inside the APC Cabinets. Originally this room used to be 100% carpet. They came in at some point in time and ripped it up and laid PVC tile, however they skipped anything that had stuff in front of it, thus you see the ugly carpet spots.



More cleanup. The east wall shelves have been removed. We also put the UPS on a rolling dolly so we can move it around. Some of the carpet has been ripped up to finish the floor for Phase 1. The west side of the room will remain carpet until Phase 2.



west side of the room pre phase 1.



One of the important things we needed to do was move our fiber demarc to a different location in the room to allow us to install some other equipment where the demarc used to be. Once the Qwest MUX was removed we were able to terminate our T3 contract which saved about $15k per year. We moved our PRI’s over to copper. Since Qwest gets all agitated when you touch their equipment (and because I told them unless they send people I was going to move it myself) they were nice enough to send a crew to move the fiber.



Their 3 man crew made short work of it. In and out in about 45 minutes.



The Fujitsu units going up.



The north wall. All the electrical Panduit conduit has been removed to make way for wall repair and paint. HVAC remotes seen on the right.



A/C unit compressors on the outside of the MDF. There wasn’t enough room in the caged area to put them side by side so the HVAC guys built a double decker holder out of unistrut. Rather novel solution in my opinion.



Our maintenance department laying new PVC tile for us. Took a month of pestering him, but when he started it only took about 20 minutes.



Phase 1 PVC tile laid prior to sealing and waxing. I was worried when the new tile was such a contrast to the old tile, but the maintenance guys assured me once they stripped and waxed it I wouldn’t be able to tell where the new tile was.



Phase 1 stripped and waxed. They were right! You can’t even tell. Initial ladder rack on the left.



Next is electrical upgrades in the district office MDF. We had an electrician come out because we were getting rid of the Symetra and putting in newer units. It required the 3-phase disconnect to be switched out to a panel (load center). We also took the opportunity to demo the single phase panel. The new 3 phase panel will provide all the power for the room.



The 3 phase 480v sub panel that our new panel was coming off of. This panel was a direct connection back to the main for the building. We had to shut half the building down during this part because the old electrician that wired up the 3phase disconnect didn’t pull in a neutral. We also found out that they had wired the Symetra’s 3rd phase as the neutral. A BIG no no.



Inside shot of the 3 phase disconnect that was going away. You are seeing a fuse for each phase.



3phase disconnect and single phase panel now gone, new 3 phase panel going in. He is wiring up our new dual 8Kva UPS’s. Each unit is a dedicated 50amp single phase. And they aren’t sitting on the floor this time.



Circuits wired up and ready for load.



New 3 phase panel complete and turned on. The flex you see coming out of the top and bottom is temporary. Phase 2 will bring permanent outlets.
District Office MDF Phase 1 complete.
Will post more in the next few days. :)
 
Last edited:
Teaser. A school's stage A/V upgrades.

Another separate mini project that we did was upgrade the A/V system in one of our schools. This particular school has the biggest (and newest) auditorium so it is where we host our yearly back to school kick off rally for all our teachers and staff. It’s a few hours of food, new teacher introductions and our superintendent speaking about the upcoming year.
The old A/V system was terrible and I don’t have any pictures of it. By the time I arrived to take pictures the A/V guys had already disassembled the old stuff.



The guys getting ready to rig the new 14 foot screen. We had to pull down an existing 20 foot screen that was front projection and install a 14 foot rear projection.



New screen going up



Old A/V enclosure on the left, new enclosure in the middle by the ladder. That is a 14 foot ladder by the way. One of the installers going up into the crawl space to get the rigging going for the new screen.



The “crawl space” that is above the ceiling of the stage. This picture is taken from the top of that 14 foot ladder poking your head up into an access hatch.



Replacing the speakers.



What the old speakers looked like after pulling them down. School was built in 2003 so that is 10 years of dust.



Getting the wiring paths for all the new HDMI and audio cables.



Final shot of the A/V control after install. Audio mixers, wireless MIC receivers, and the amplifier. Also on the left is a Crestron push button system that turns on the projector and lowers the screen at the same time.



The rear projector. Also seen is our IP SIP based intercom paging system



What the students and staff see.
 
Nice stuff. Those Crown amps are excellent! Some of that conduit work leaves a bit to be desired though...
 
jeffmoss26 said:
Some of that conduit work leaves a bit to be desired though...
Click to expand...
I know. But the electricians were pissed off at me because I had already made them re-do the conduit work at the top of the unit twice because they kept doing stupid stuff. At that point I told them just finish and get out.
 
Ok. Massive update time. My posts will be rolling out much more rapidly now that all the pictures have been taken and cleaned up.

I’m going to save our district office MDF (seen in my first post) until last, since it probably has the most dramatic renovations.

Let’s start at one of our schools. This particular school is the same one that had the A/V upgrades which you saw in my last post.



Let’s start things off with how this MDF used to look. You see 2 racks in the center, these are the server racks. This school had a VERY old Cisco 4000 chassis which was layer 2 only. They had another 3600 router that did all the Layer 3 routing. Whenever we would image at this school it would take the routers CPU to like 90%. On the right you see A/V racks for the old campus video distribution. These were literally VCR’s and Coax amplifiers.
You also see on the left the EMS (energy management system) PC that controlled all the HVAC at the school. This has since been upgraded to a district wide server and can be centrally managed. (And finally a single slot LTO3 tape drive that they never used)



A shot of the old and new during transition. The servers (all 1 of them) was consolidated into the network rack so we could remove 1 of them to make room for the new 2 post racks. Seen is Cisco 4000 inside the network rack and the ugly cable management.



Beginning to pull all new cable in the school. All told, across our 5 schools over 500,000 feet of new copper was run.



All finished! On the right you might notice that we closed in the gap in the wall to make the room more secure.
New Cisco 4500E with all gigabit copper blades. New Cisco 2921 voice router for SRST. They also contain all the DSP’s and FXO/FXS cards. And a new(ish) HP G6 server to run the VM’s for that campus. (Most VM’s are centralized in the district office’s Hyper-V cluster, but each school has a few site specific VM’s)
On the right 2 post rack you’ll also see our new amplifier that powers the remaining analog speakers for the intercom/paging system. 80% of the system is IP based (as seen in previous pictures), but there are still common areas where it is cheaper to keep the existing analog. (Hallways, bathrooms, exterior, etc)
All the coax is now wrapped up in a nice loop because our TV distribution is now all digital and runs over our IP network. No more coax amplifiers and VCR’s.



A rear shot. No cables on the floor. That was a requirement of mine that nothing be draped across the floor. This way you can walk completely around all our racks without tripping on anything.



Top view of the new room. Nice cables bundles.



A close up of our new fiber labeling. Everything is now properly labeled and documented. I wrote an 80 page network and systems documentation binder so we know where everything is and how it’s connected to each other. We also now have all our servers documented with proper IP’s, and what roles and services they all host.




-----------------------------------------------------------------------------------------------------------------------------------------------------------------





Here you see this schools 1 and only IDF. All this stuff in the hallways was what I had to clear out of the room before we could begin renovations. The schools think that our server rooms are their storage closets. We had to resort to completely re-keying all of our MDF’s and IDF’s throughout the district to lock them out and prevent them from filling up our areas again after the renovations were completed.



What the room looked like after we did an initial round of cable management and cleaning about 6 months before renovations started.



The IDF now. Yellow cables on the right are the access control vendor pulling in composite cable to get our exterior doors secured. All schools in the country are pushing hard for this new safety and security because of all the recent violence.
 
jeffmoss26 said:
Looks great! What is all the wiring on the back wall coming out of the 3 pipes? Looks like intercom stuff maybe? http://i.imgur.com/941FNnF.jpg
Click to expand...
The analog portion of the intercom system.
Above the amplifier in the 2 post rack you will see 2 devices that have ethernet cables coming out of them. Those are our "zone controllers" for the analog speakers. It translates IP SIP to analog, which then goes to the amp. Coming off of the amp we run a cable over to the 66 blocks which distribute the analog signal out to the individual speakers throughout the school.

The IP speakers are paged directly via SIP
 
Wow this is some hardcore porn right here! Great pics!

It feels great to do a cleanup like this and get to redo stuff new, and properly. Did my home server room recently, obviously nothing even close to this magnitude though.
 
Ok. Another update. This time at a different school. What makes this school’s MDF unique is that we moved it. The original MDF was too small to support more than 1 rack, so we (IT) took over an office that is on the other side of the wall of the existing MDF and transformed it into the new MDF. Since all the copper and fiber was being re-pulled in the campus anyway, it was actually easier to transition because we weren’t dealing with having to run an old and a new MDF in the same room at the same time.



Old MDF before being cleaned out. The cabling estimator can be seen taking pictures so they could finalize the bid to re-pull all the copper and fiber.



After cleaning the old MDF. It sat in this state for about a year while all the bids were finalized and money was allocated for the project.



The “office” we took over before we began renovations. The room is about 15x15.



Phase 1 complete. Carpet replaced with VCT tile, white board removed, and room painted.



Phase 2 complete and in production. New 1 ton mini-split system added, access control panels, storage shelves, and of course the district’s standard rack layout.


------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------



A very rare shot indeed. One of our only indoor IDF’s. This is another instance of the old room being too small for a rack so we moved the IDF into a storage room off of a computer lab.



Behind the rack. Grounding, and cable feeds.



A shot of the lab this IDF is in. I found this picture ironic.
 

A shot of the lab this IDF is in. I found this picture ironic.[/QUOTE]


Not sure if you are pointing out the aged computer basics with what appears to be a p3 era dell with the 80's keyboard and old apple OS, or the floppy drive in the optiplex :p
 
I think you should buy that guy taking the pictures of the cabling a donut.......dude is going to drop dead of a heart attack if he holds his arms up too long!!:eek:

Great pictures. Must be a really expensive project!
 
Alright. Time for another update. This one will be yet another school. In this school we kept the MDF in the same room. In my previous posts I had mentioned about our old A/V cabinets that were literally VCR’s and coax amps. Well here is a picture of what I was talking about that was in each MDF’s before we renovated. It took up the bulk of the space in each room.



What the MDF looked like prior to any cleanup. On the left is the 1 and only server rack with an old model 4500 (non E series) and a UPS. The other 3 racks are A/V and intercom. The middle 2 racks are the VCR’s, coax amps, and modulators. The old Apple was some kind of video converter server. By the time I started working here it had already been abandoned for several years. The right most rack is the old full analog intercom system. It was installed in 1997 and to its credit it had been chugging along for about 16 years without incident. They built those things to last.



Behind all the A/V racks. You see the old NEC PBX phone switches and all the 66 blocks that were required to run that system. Everything on the floor isn’t being used. Yes that is a UPS sitting on top of a bucket if you look close.



The nappy floor after all the boxes were removed.



Renovations complete. There was a door where the map is on the left. But we had it removed. It led to a computer lab but the school kept refusing to ever lock the door. So we removed the issue by eliminating the door and locking them out. As part of renovations with doors and painting we also had the construction team polish and wax all the floors when they were done. Gives them a nice shine.



The rear of the room now. Any 66 blocks remaining go to powering the analog portion of the intercom system. You may or may not be able to tell, but a LOT of copper was removed from those 66 blocks since we only have a handful of analog speakers now.



The conduits and floor after cleanup.
 
Those Rauland systems are very well made, but they are proprietary and only have one dealer in a given area of each state.
 
Let me know if you want to ship that Mac G3 AIW box, those things are sweet lol!
 
Update time. Yet another completely different school. This school was unique because the MDF was on the second floor. We also had to commandeer another new room to rebuild our MDF. The original one was completely exposed in the library and didn’t even have walls. It was just racks sitting in a back corner.



Old MDF. I was at least able to find the old keys for the network cabinets and lock them so little hands didn’t find their way into our switching gear, but it was a terrible spot for an MDF when they remodeled the school back in 1997. Now that we had the resources, it was time to move it.



New MDF room prior to renovations. This was originally a study hall type room off the main library. The “old” MDF area is around the corner on the other side of an elevator shaft.



New MDF after Phase 1. New floor, ripped out all the mill work and painted the room. We also had to build a wall that you can’t see to isolate the room from the rest of the study hall room.



New MDF in production. Moved the fiber to this new room, pulled all new copper throughout the school, and also did electrical upgrades to support 280v circuits in this room.



An example of our new initiative to label everything. If our Metro-E circuit ever goes down we have the NOC number right on the switch so there isn’t any time wasted figuring out who to call. And with circuit ID’s on everything it’s finally easy to identify what is what. This all may seem like common sense (and it is) but nothing was labeled prior to me working here so it took forever to figure things out and what connected to what and where. Now it’s all documented in 87 pages of clean network drawings.


------------------------------------------------------------------------------------------------------------------------------------------------------------------------



One of the IDF’s on this campus after remodel. A typical IDF in the district.



A typical computer lab in our district. This one is bigger than normal, but they all pretty much look like this.



My customizations for WDS imaging. The techs love the 1 button imaging I setup, and our students and staff like the look of WinPE since I made it look pretty. Their fascinated with what our techs are doing when we start the re-imaging.
 
Nice work, very nice that you get to do a complete overhaul.

I have usually had to do partials, and you can never get it to look as nice or documented out as cleanly as you want.
 
You must log in or register to reply here.
Back
Top