We were trying to finish up a major email blast tonight and all of a sudden our main blast server stopped working.
Then I immediately got a flood of email alerts saying all of our live servers were down.
I submitted a trouble ticket to Amazon EC2. I have Gold Support package lately with all the email issues we have been having.
I looked at their AWS Status Page and it reported nothing. Then shortly after I submitted my ticket I see they refreshed the status:
Power issues in a single availability zone
Just the East region was having issues. Our instances in the West zone were fine. This was at the same time they were providing PTR records for our Amazon EC2 email server IP addresses.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=14c028e0-a643-441d-a059-02d25e9cc494)
Tags: Amazon EC2, Amazon Web Services, Cloud Computing, Servers
We also experienced problems, to put it mildly. Two of our EC2 servers had been automatically rebooted. This killed some of the changes we had made to the instances after they had been launched.
This taught a lesson to us – never make custom changes in an EC2 instance after bootup. Always burn your changes into a new AMI and launch the production instances from that AMI.
If there any run-time configurations, put them into the init.d scripts before burning the AMIs…
Now we are in the process of creating production AMIs from our running instances and we will re-launch new instances from those AMIs…
I agree about your lessons learned. The cloud teaches us a new level of approaching system administration.
I have related it closer to actual software development. I have talked about a cloud version control system for managing this. Help you manage changes made to AMI….then deployment of these AMI. Keep logs, rollbacks, etc.
Definitely having a process helps you slow this process down and be more mindful of changes you make.
Nothing is permanent in the clouds. Cover your ass.