If you've never had an opportunity to spend time in a datacenter as a software developer, that's unfortunate but also far too common. What things look like on the inside vary company to company. Generally you're in an OSHA-abiding environment, so safety shoes, ear and eye protection, sometimes gloves.<p>There's a variety of roles. Security, electricians, HVAC engineers, generally some type of site foreman-ask role, logistics (depending on the size of the place), and technicians (for a lack of a better word, feels like every place calls them something different). There's a variety of roles that often float between sites or oversee many sites, depending again on the scale of the place. AWS is huge. Bigger than you're imagining, so there's quite a few levels deep and include real estate folks as well as construction roles. If you go and look at job postings, you'll even see roles for nuclear engineers at some companies.<p>But generally what you're talking about here are what I'm calling the technicians. They're responsible for wheeling racks into place (depending on the company they may also be responsible for unloading the trucks). Cabling is nearly always outsourced these days (though not the design of the cables), so rolling a rack into place generally involves securing it to the floor and connecting power, data, and more often than not now-a-days liquid cooling.<p>The other part of their job is "troubleshooting" failed hardware. Again, really depends on the company. Big big shops have "dumbed down" troubleshooting as much as they can - for a lot of reasons. You don't have to pay folks as much because they're thinking and doing less, the more time they spend troubleshooting the longer the server is offline, and if there's no troubleshooting there's not much for them to screw up. I'm sure there are some great places to be a tech where you get to rip apart servers and bust out the multimeter, that to my understanding is not how the hyperscalers who actually hyper-scale do it.<p>There's some cleaning, parts management, destroying broken hard drives, shoveling snow off the roof (no lie), and a variety of other odd tasks.<p>If you ever have the opportunity to check out one of those places it can be a riot and a real eye opener. Depends again on the company though, some of those places have insane security (metal detectors, badge+pin, turnstile door procedures) which make visits super un-fun if they're even allowed outside of legit business reasons. Other companies... well I'm glad that's not where I store my data.<p>Back "in the day" (2005 give or take a handful of years) techs would often write their own automation and even build some simple services.<p>And yes, the jobs don't pay particularly well depending upon what it is. Electricians and such command decent wages, but the security guards and techs don't make crazy amounts. I think folks doing contract cabling can come out ahead.<p>Anyhow, SWEs are wildly insulated from the realities of what things look like on the ground. Maybe that's a good thing, IDK.
> Big big shops have "dumbed down" troubleshooting as much as they can - for a lot of reasons. You don't have to pay folks as much because they're thinking and doing less, the more time they spend troubleshooting the longer the server is offline, and if there's no troubleshooting there's not much for them to screw up.<p>Very true. I've heard stories of how technicians struggle with friends/family perceptions here. Since a lot of these datacenters are in rural communities, they are perceived as being technical wizards to be working there. But in reality they are doing as you say - just following a preprogrammed script with very little scope for any sort of creative problem solving.
Not the poster, but I highly recommend checking out your company's data center if you can (and if your company has one). Nothing like seeing where your apps or data actually land (and the physical security put in place to protect them)<p>You'll especially be in luck if your company is an old and has a mainframe or two. Those are incredible to behold. Masterful engineering.