August 18, 2008
Here's the update from Stanford
Power mostly restored to campus
Updated 3 p.m pacific time. - Power has been restored to the majority of the Stanford campus. All normal power is expected to be restored by 5 p.m. or sooner. A major outage occurred today at 11:30 a.m., affecting P G & E customers in Stanford, Menlo Park, Atherton and Palo Alto. If you are still experiencing difficulties on the Stanford campus, call 723-2281. At Stanford Hospital, dial 286.
This one was a major disaster at Stanford and Palo Alto (and nearby cities), probably the biggest outage in a while. However, there seems to be some major Stanford power outage once a year, which is a major problem.
With this in mind, we have been distributing more of FAH to outside of Stanford (with servers at UCSF, Columbia, Cal State U Long Beach, and U. Pittsburgh). We hope to have a European site soon. Once those sites are a bit more established, we'll see about pushing an assignment server to a non-Stanford site and we should be much more safe to Stanford-related issues.
Also, it's good that we have servers in 4 different server rooms on campus. One stayed up the whole time, two came up fast, and the fourth (VSPGxx) is coming up now. Some servers will be slow to come up, so we expect this may take at least a few hours.
The stats update has been turned off until this gets sorted out. We hope to turn it back on later tonight, but it may have to wait until tomorrow morning.