So what does one of these full time data center jobs look like, day to day? If I’m a software engineer I feel like I’d have to move and get a pay decrease to actually work at one of these? I mean until AI finally puts me out of a job. I guess I wouldn’t really be qualified to work one of these jobs?
The blokes I meet who work 8x5 from a data center tend to spend their days installing hardware, deinstalling hardware, providing remote hands (usually cabling, sometimes console access)building racks, managing power supplies and maintaining asset registers. And escorting idiots like me around when things get technical.
If you've never had an opportunity to spend time in a datacenter as a software developer, that's unfortunate but also far too common. What things look like on the inside vary company to company. Generally you're in an OSHA-abiding environment, so safety shoes, ear and eye protection, sometimes gloves.
There's a variety of roles. Security, electricians, HVAC engineers, generally some type of site foreman-ask role, logistics (depending on the size of the place), and technicians (for a lack of a better word, feels like every place calls them something different). There's a variety of roles that often float between sites or oversee many sites, depending again on the scale of the place. AWS is huge. Bigger than you're imagining, so there's quite a few levels deep and include real estate folks as well as construction roles. If you go and look at job postings, you'll even see roles for nuclear engineers at some companies.
But generally what you're talking about here are what I'm calling the technicians. They're responsible for wheeling racks into place (depending on the company they may also be responsible for unloading the trucks). Cabling is nearly always outsourced these days (though not the design of the cables), so rolling a rack into place generally involves securing it to the floor and connecting power, data, and more often than not now-a-days liquid cooling.
The other part of their job is "troubleshooting" failed hardware. Again, really depends on the company. Big big shops have "dumbed down" troubleshooting as much as they can - for a lot of reasons. You don't have to pay folks as much because they're thinking and doing less, the more time they spend troubleshooting the longer the server is offline, and if there's no troubleshooting there's not much for them to screw up. I'm sure there are some great places to be a tech where you get to rip apart servers and bust out the multimeter, that to my understanding is not how the hyperscalers who actually hyper-scale do it.
There's some cleaning, parts management, destroying broken hard drives, shoveling snow off the roof (no lie), and a variety of other odd tasks.
If you ever have the opportunity to check out one of those places it can be a riot and a real eye opener. Depends again on the company though, some of those places have insane security (metal detectors, badge+pin, turnstile door procedures) which make visits super un-fun if they're even allowed outside of legit business reasons. Other companies... well I'm glad that's not where I store my data.
Back "in the day" (2005 give or take a handful of years) techs would often write their own automation and even build some simple services.
And yes, the jobs don't pay particularly well depending upon what it is. Electricians and such command decent wages, but the security guards and techs don't make crazy amounts. I think folks doing contract cabling can come out ahead.
Anyhow, SWEs are wildly insulated from the realities of what things look like on the ground. Maybe that's a good thing, IDK.
Not the poster, but I highly recommend checking out your company's data center if you can (and if your company has one). Nothing like seeing where your apps or data actually land (and the physical security put in place to protect them)
You'll especially be in luck if your company is an old and has a mainframe or two. Those are incredible to behold. Masterful engineering.
> Big big shops have "dumbed down" troubleshooting as much as they can - for a lot of reasons. You don't have to pay folks as much because they're thinking and doing less, the more time they spend troubleshooting the longer the server is offline, and if there's no troubleshooting there's not much for them to screw up.
Very true. I've heard stories of how technicians struggle with friends/family perceptions here. Since a lot of these datacenters are in rural communities, they are perceived as being technical wizards to be working there. But in reality they are doing as you say - just following a preprogrammed script with very little scope for any sort of creative problem solving.
You're likely unqualified. You'll be competing against a pool of people who are qualified and have worked these jobs.
I used to do SWE. Early in my career I realized that DCO/DCE was not only an entire discipline unto itself, but that the rest of the "team" looked down on it because it required physical labor. Perfect, I don't feel like I've done Real Work unless I do something in the physical world. Hence my DCE jobs. I would not be able to do SWE today.
But I'll tell you this: way more than once I've had a cocky new hire from the HFT I work at wither under simple tasks: Please re-balance the load across all phases of these PDUs. Please groom & remove dead cables from wherever. But you don't have time to figure it out on the fly. You have an hour to have it DONE.
Anyway, I think your intuition serves you well. :)
That's startup to mid sized traditional company thinking. Not at the hyperscaler enterprise scale.
Anybody working in even classic datacenter physical ops already knows how to plug a KVM with a cell modem into a box to let the engineers remote in. That's assuming the racks aren't already built to support this natively these days.
Come on, this is the industry that is going gangbusters on the fetish of mass unemployment and deskilling, you don't think they're doing everything they can to have to only hire a few local bodies at minimum wage to basically pull a bad rack out and slot a new spare in?
It's not easy, actually. You will likely need to be a licensed electrician or a licensed plumber. Both occupations require around 4000 hours of apprenticeship.
Some states don't need a license for low-voltage work, so you might be able to do data wiring.
I started reading the Camp of the Saints precisely because people said I shouldn’t. It was a bad book, I couldn’t read more than a few chapters. But I think adults should be able to read whatever they want.
The 'curated collection in a light bulb' in your strawman fallacy contains books you could find prominently displayed in every mainstream bookstore's entranceway. Hardly banned by any reasonable definition. Pilpul not withstanding.
The opposite of 'banned books' making it false advertising.
I live in the suburbs of Kansas City where the World Cup is happening, and the only way it’s affected me is it’s really annoying to take my disabled kid to the children’s hospital downtown because there’s so much traffic.
I know exactly one person out of my friends that is going to one World Cup game, and he’s well known as the guy who likes soccer.
My wife has enjoyed the TikToks of Europeans coming to Kansas City and reacting to stuff that’s totally normal to us, like the Bass Pro Shop and Tornado Shelter signs.
I’m using $200 a month Codex working on a game for my kids for fun and curiosity since I’m a dev, I’ve played games, but I’ve never done dev for games. and have all night tasks but mostly they’re “spend time tending to and adding stuff to my 3D asset pipeline”. My RTX 5090 runs Trellis2 -> ultrashapes -> Trellis2 -> wiring up rigging and setting up animations.
But like 99% of that task is just Codex waiting for the output. So it’ll run for 12 hours but mostly it’s just setting lots of sleeps. I haven’t gotten close to running out of tokens. The $100 a month codex I hit usage limitations almost immediately, about 3 days in of working like crazy with 10 agents going at once, mostly coding an asset pipeline, I ran into my weekly limit and upgraded. So with the $200 a month plan at 4x more credits I haven’t hit any walls at all and can absolutely cook.
This sounds like you're overcomplicating things a lot and like you're very unlikely to be learning anything useful, I would suggest making something simple yourself to get a handle on what making the different parts of a game actually means in practice.
Knowing LLMs and their output I would also bet that you're getting nonsense output that sucks.
Yeah I’ve been working on a game just for my kids, it convinced me to upgrade to Codex Pro and I absolutely wouldn’t release it to anyone until I felt 100% it actually was fun to play. It’s easy to get stuck not doing the stuff that can’t be automated. The crazy thing is that making the game pretty good looking (like Nintendo Switch level graphics) is basically trivial now and can be largely automated, maybe with a little Blender cleanup for your most important assets. That doesn’t make a game fun though.
I ran into this wall too. Someone here on HN said their general test is “make a browser only simple rts game with no AI and no multiplayer” What makes a game fun is very different than the engine of a game. My kid asked me to make a game where you brew potions. Okay, done. Adding ingredients, having physics to drop the items rather than “okay they just appear in the pot” (or worse it just says “okay they’re in there!”) Especially for kids there’s a physicality you have to capture to make it both fun and understandable to a seven year old.
Getting cutesie stylized 3D models is something that’s trivial with an RTX 5090, a ChatGPT pro subscription (unlimited image generation), you run Trellis2 plus a few other open source things in a pipeline that your agents can queue and it’s astonishing how much cool stuff comes out the other side. But the graphics don’t make the game fun at all, they’re just set dressing for the fun.
There’s been a lot of learning going from 0. “Okay, 3D model of a character. Oh, this model is useless since it isn’t in a T Pose I can’t rig it. What’s a rig? Okay, there’s a rigging ML model. Download that. Okay, how do I animate it? Oh, cool there’s a model for that. Oh wait my model has holes in it, that looks weird. Okay there’s an ultra shapes library that helps improve geometry. Whoops, that strips all the textures and shaders. Okay, trellis2 has a mode that takes an existing model and retextures it. Okay wow these look good, the characters are walking around! This goblin is break dancing! Okay uhh, what do you actually do in this game?”
Like it feels like that trap you can get stuck in when one part of something is trivially easy, so I have like 500 random 3D assets that are honestly pretty good looking for a game where the core gameplay loop is not developed at all because I have no idea what would make it feel fun. Because I can prompt and say “oh wouldn’t a Christmas village be cool?” And I wake up the next morning with 50 3D models of Christmas village stuff and characters and I say “wow, neat!” (It takes maybe 8 minutes end to end for the pipeline to generate one 3D model, so I just run it overnight). But then I have to manually place them in the world (if you let the AI do it in unreal engine 5 it places them via coordinates which become impossible to move inside the unreal editor).
The fun part is “wow, this is something I’m making with my kids, and it’s unique to us”. That’s what keeps me at it. I’ve never seen my kid so engrossed and excited to help me with something, she’s the one coming up with ideas and saying “what if we did this and that?” and then seeing those things become real is really neat. The bottleneck is there’s a dozen agents that can work on different parts of the game but it’s a chaotic mess.
Still, I’d imagine this is how people learn is by making something that’s a piece of junk then making something that’s better. I don’t plan on releasing my pieces of junk unless I feel like they’re actually fun.
I remember working for a company that did a lot of business in logistics. We were strictly prohibited from using any Amazon Web Services because several of our very high profile customers didn’t want anything on AWS. The higher ups were thoroughly convinced Amazon would copy it (and I mean, they came out with a product that competed with us, so they weren’t wrong!)
This kind of stuff “but they’ll copy us” is always weird (and wrong). Logistics isn’t some secret sauce. It’s taught in operations degrees across colleges. If a company is worried that all it takes is another company “copying” their IP to supersede them, then you don’t have a company, you have a simple app.
Amazon didn’t “copy” logistics from Apple. But both of them use similar underlying processes and optimizations. They both excel at it, and neither is eating the other’s profits. The same goes for smaller companies. Or the logistics providers like UPS.
My wife makes real Eggs Benedict for me once a year on my birthday. I went to a resort hotel and spit out their Eggs Benedict it tasted so bad compared to what I get at home. They comped my meal. I guess this explains why I’m constantly confused why Eggs Benedict doesn’t taste right. I’ve found one, maybe two restaurants with passable Eggs Benedict in my city.
reply