A massive power failure hit downtown Vancouver this morning, the result of a fire at underground power lines under Richards St between West Pender and Dunsmuir St. Power has been out for buildings and traffic lights around the downtown core. Downtown traffic is hectic at the best of times, and completely gridlocks in the absence of functioning traffic lights. The underground nature of the fire has made repairs difficult and as late as 3pm it was still not safe for BC Hydro crews due to poor air quality underground.
Noticeable across the net was the loss of the Vancouver Peer1 datacenter in Harbour Centre, which hosts popular Vancouver sites such as Plentyoffish, Suite101, Hushmail, and Bravenet. Apparently the building's backup generators overheated and failed, and there were witness reports of smoke being vented out of the building (which is some distance from where the first). I've recieved an update that Peer1 is powering servers back on at this moment, so these sites may (or may not) be back shortly. A thread on the Peer1 forums is being regularly updated. Techvibes Blog is hosted by Peer1 subsidiary Serverbeach in their San Antonio datacenter.
More updates and pictures from NowPublic.
Update: Ian Bell posts an explanatory email from Peer1 in the comments below. The jist of the failure was that the backup generator came online at the moment of failure, and worked for 20 minutes before failing. The generator was water cooled, and as firefighters responded to the nearby fire, water pressure dropped causing the generator to overheat.
Looks like everything is back up now (as of 5h30pm) but Peer1 is running on Diesel fuel - let's hope that the real power comes back very soon.
Apperantly power will be restored this afternoon, according to cbc!
Nice heads-up response from Peer1 CEO Fabio Banducci, however Harbour Centre is a ridiculous place to have a CoLo from a power perspective. The building is tapped out -- in fact, bursting at the seams, due to its inability to supply power. Peer1 needs to address this by finding new, less encumbered, real-estate.
Email below:
___________________________
Dear PEER 1 valued customer,
I would like to take this opportunity to sincerely apologize for today's disruption to your business. We understand fully that you depend on PEER 1 as your trusted IT infrastructure provider to ensure your online business and applications remain online all of the time. In this instance, we failed to deliver on this promise. We will, of course, fully honor our service level agreement commitments. We also commit to conduct a thorough post mortem review of the events of today and take all measures necessary to ensure that your online presence is protected and not disrupted again in this fashion.
Please allow me to give you a brief rundown of today's events. At 9 AM today, BC Hydro reported that a number of electrical circuits failed in downtown Vancouver, British Columbia due to an underground fire on Richards Street that caused a power outage to approximately 2,200 customers, including the Harbour Centre building where PEER 1's Vancouver Data Centers are located. BC Hydro crews began working to determine the extent of the damage, but had difficulty accessing the man-hole where the fire originated due to extensive smoke and heat.
Harbour Centre's main building generator, Gen No.7, started and maintained power as designed, but failed 20 minutes after the loss of commercial power due to a failure in the generator's cooling system. Gen No.7's cooling system is dependant upon water provided by the city of Vancouver and requires a certain level of water pressure to function normally. As fire fighters worked to put out the fire that had broken out on Richards Street, water pressure in the downtown core, (including Harbour Center), declined to a point where it caused the generator to overheat and malfunction.
Once the generator failed, PEER 1's UPS systems eventually failed as well once the UPS batteries had drained. After three more failed attempts to bring Gen No.7 online, the generator successfully came online at 3:40 PM and restored power back to our Data Centers, our UPS systems, and our client servers. At 5:55 PM, utility power was fully restored to the Data Centers, and Gen No.7 was shut down. A detailed timeline of events has been provided to you at the bottom of this letter.
Please do not hesitate to contact me directly with any questions or concerns that you may have. My direct telephone number is tel: (604) 909-6424.
Thank you for your patience and understanding.
Sincerely,
Fabio M. Banducci
President and CEO
PEER 1
TIMELINE
9:00 AM - Fire started in a manhole located in the 500 block of Richards Street, Vancouver, BC.
10:00 AM - Utility power to most of downtown was lost due to a subsequent transformer explosion. Harbour Centre's main building generator, Gen No.7 started and provided power to the Data Centers. No clients were affected.
10:20 AM - Gen No.7 failed due to overheating as the result of a failure in the generator's cooling system. Power went with UPS systems now running on battery.
10:40 AM - First alarm of batteries failing in UPS systems.
10:45 AM - 11:30 AM - Clients lost power due to UPS batteries running out.
12:30 PM - Gen No.7 is restored and providing power to the Data Centers.
12:30 PM - 12:45 PM - Clients' servers are back up and running.
1:00 PM - Gen No.7 fails a second time due to overheating as the result of a failure in the generator's cooling system. Clients lost power within minutes due to UPS system batteries being low.
3:00 PM - Gen No.7 is restored and providing power to the Data Centers. PEER 1 did not turn client servers on immediately; waited for power to first stabilize.
3:30 PM - Gen No.7 fails a third time due to overheating as the result of a failure in the generator's cooling system.
3:40 PM - Gen No.7 is restored and providing power to the Data Centers.
4:00 PM - Power restored to UPS systems, but not to client servers; waited for power to first stabilize.
4:20 PM - ALL UPS systems now online and power to client servers restored.
5:55 PM - Utility power restored to the Data Center and Gen No.7 shut down.
[...] content management/blog system currently in beta. After a slight delay due to this week’s downtown Vancouver power outage, they’ve launched a new build adding a new toolbar, status bar, social bookmarking links, [...]
