Note on Opinions
They are mine!
Inside Cumulonimbus
AWS is enormous. However, the term ‘enormous’ doesn’t quite capture the scale I’m referring to. Around 2002, Jeff Bezos mandated that all teams expose their services through interfaces that could be accessed internally or externally. The impact of that mandate was palpable in my day-to-day job because there is literally a service for every imaginable function. Need an internal user directory? There’s a service for that. HR portal? Another service. DNS Management? Several services to choose from. Booking business travel? An internal service handles that. Whiteboarding? Yep, there’s a service for that too 🤖. The sheer breadth of services can be overwhelming, and I’m sure end users felt it too.
I’m not suggesting that a company’s size or influence is solely determined by the number of exposed services or APIs. What I am saying is that the multitude of services I interacted with daily (likely a small fraction of the total) was staggering.
Navigating AWS (Identity)
I spent a lot of time focusing on Systems Programming and other related courses in college with the hope of working on those systems. When I received an offer to work at AWS, I eagerly accepted, hoping to be placed on a team involved in the development of Amazon Linux or Route53. In August 2020, I found out that I would be joining the AWS Single Sign-On team within the AWS Identity organization. My manager at the time admitted that he, too, ‘fell into Identity,’ but it was well worth it. Even though the applications I worked on were not highly related to systems, I did learn a lot about Identity and building services.
Process
I felt that my team was still figuring out the right balance between process and letting people… work. I cannot say I had a perfect reputation (read: I broke production once in a big way!) but I was one of the most productive members of my team. Part of that is because of my teammates were supportive and the other part is that I wasn’t afraid to break things. Granted, it took me some time to figure out how to break things safely. In general, you will find that many AWS teams have many long processes related to getting changes out the door. The biggest reason, in my eyes, is that bad changes have a huge impact on the internet at large. From the perspective of shareholders, that is a good thing!
I do wish more of the “lower ranking” (think L4 and L5) team members were included in the meetings that decided process. Unfortunately, things were moving so fast towards the end of my tenure that I felt like I was being left in the dark. That isn’t a great feeling because it means you have less to contribute if you aren’t “in the know”.
Code
Believe it or not, I didn’t find myself coding as much as I thought I would. It was still a critical part of the job, but I spent the majority of my time writing and planning. In the last year or so, I worked extensively with the AWS CDK (IaC) and built some internal tools for our team using Golang. In general, the technology you work with depends on the time period in which your company was created. For example, Ruby on Rails was created in 2004. I can assure you that companies started or growing around that time (e.g., AWS, GitHub) use Ruby on Rails for their services, web apps, etc
Oncall
Your on-call experience will highly depend on:
- The product
- How often it is used and when it is used
- Where your product sits in your organization’s roadmap
In my case, this was my breakdown:
- A Single Sign-On platform
- All the time. Customer and non-customer traffic were always being routed into the Control Plane.
- The AWS SSO (now IAM Identity Center) offering is going to end up being the front door to human access into AWS systems and other applications.
That third point is critical. Being the front door to any major cloud means you cannot be unavailable. With that said, I both appreciated and disliked on-call. Getting paged at 3 A.M. and checking into a ticket to find that a latency threshold had temporarily been breached was not fun. On the flip side, getting your hands dirty with on-call is one of the best ways to learn the systems you are responsible for and understand your customers’ pain points.
Generally speaking, AWS pursues a high bar for monitoring and maintaining anything they provide to customers. That includes internal customers (e.g., my old teammates). While doing ops isn’t always fun, it is a good learning experience that most developers can use to grow. That is, until the industry eventually repeats having dedicated QA and sysadmin teams. I hope not, but many ideas get created, then repackaged and ‘recreated’.
Preparation
There isn’t any one book or collection of books you could read to prepare. What makes the most sense, at least in my head, is to focus on sharpening your interviewing skills and casting a broad net in terms of the jobs you will apply for. I applied to roughly 200 different positions during my senior year and many more when I was looking for an internship.
When you land your first job, have an open mind about the work. It might not lead to the sexiest elevator pitch, but that is okay since you’ll likely just be learning.
Final Thoughts
Wherever you land, I recommend understanding your manager’s priorities and helping them make good on their commitments. If you have the luxury of choosing whether you want to work somewhere, then I highly recommend leaving, switching teams, organizations, etc. if you and your manager don’t get along well. I will die on the hill knowing that managers are hugely responsible for your career since they are the first points of contact for everything you will do. Promo? Well, let’s see your manager’s feedback. More interesting work? Your manager assigns work to your team. I have had my fair share of good and bad managers at Amazon, and I know enough to stick up for myself. It can be easy to forget that people will roll over you if you let them.
When you onboard, take all the time you can to learn. You might be eager to contribute, and I promise you will, but it’s harder to make time for learning when you are given more responsibility. Not impossible, just harder.
Don’t take yourself too seriously, and never marry anything you put out. The people that are the easiest and most fun to work with are the ones that are open-minded and willing to give and take feedback. Your first design document will probably be torn to shreds by your design reviewers, and that is okay. The second will make it out with lots of work to improve it. The third will be a little better. That cycle continues.
Write. AWS or not, many decisions are written down and reviewed all the time. If you do write, you will look more mature and create a trail of artifacts that you can point to later, whether for reference or promotion.
Remember that it’s just a job. Ultimately, you are building something for your end users. You shouldn’t get swallowed up by the work; set boundaries with it. E.g., if you are paid to work 8-hour shifts for 5 days a week, do not work a lick more. That is already 23% of the time you have in a week, and that is not including the fact that you are asleep for almost the same amount of time. Nothing is worth sacrificing your health and well-being for. Now, not everyone has the luxury of setting boundaries like that. If you are working on getting a promotion, you are probably working longer hours. In that case, boundaries could look like taking vacations where you don’t touch your work email or laptop and delete anything remotely related to work from your devices. I deleted Slack and our pager app from my phone when I was last on vacation. I have lots of other opinions on what “work” means, but this isn’t that kind of blog.