Doing oncall SA work has its moments but can be dull.
I have learnt the following so far:
- The night sky is always clearest at around 3am
- FSCK’s always need to be forced at around 2am
- Security at the datacenter will always make your life harder
- Clients will only notice outages (despite monitoring alerts) around 6 hours after they occurr, then panic
- Tasks always take 3 times as long due to the datacenter being timeless (constant sound and light conditions)
- RAID arrays will always throw the disk after you fixed the rest of the issues
- There is never a seat when you need one, and the floor is cold
