Plan: Transitioning to AI Infrastructure Engineer

AI infrastructure is where backend engineering meets the GPU era. Unlike applied ML (where you need model intuition) or research engineering (where you need math), AI infra roles are fundamentally systems engineering roles — with GPUs as the constrained resource, models as the workload, and latency/throughput/cost as the optimization axes you already understand.

This makes AI infra the shortest transition path for strong backend engineers. The gap is narrower than it looks.

Who this is for

A backend engineer (IC3–IC5) with experience in distributed systems, infrastructure, or platform engineering. You’ve built or operated services at scale — load balancers, job schedulers, data pipelines, container orchestration. You want to move into AI Infrastructure (inference platforms, training orchestration, GPU cluster management, ML platform teams) within 4–8 months.

What this plan assumes

• Baseline. Distributed systems design, container orchestration (Kubernetes), at least one systems language (Go, Rust, C++) or strong Python. Experience with metrics, alerting, and capacity planning.
• Time commitment. ~150 hours total across 4–6 months at ~8 hrs/week. Compressed: 8 weeks at 20 hrs/week.
• Goal. This is a plan to become employable on an AI infrastructure team, not to become an ML researcher who also knows infrastructure. The distinction matters: you need enough ML intuition to understand what you’re serving and why the workload is unusual, but you don’t need to build models yourself.
• Scope. Three surfaces: GPU systems knowledge, ML workload understanding, and a hands-on infra project.
• Out of scope. Model development, training from scratch, ML theory beyond what’s needed to understand the workload, and research engineering.

The core positioning: AI infra is distributed systems engineering where the resources are GPUs, the workloads are models, and the constraints are memory bandwidth and batch efficiency instead of CPU and disk I/O. Your existing mental models transfer; you need to learn the new vocabulary and the workload-specific constraints.

The three surfaces

The plan

Four phases. Shorter than the MLE transition because your existing skills cover more of the surface.

Phase 0: Orient (~3 days, ~6 hours)

• Understand the landscape. AI infra teams work on: inference serving platforms (vLLM, TensorRT-LLM), training orchestration (distributed training, checkpointing), GPU cluster management (scheduling, multi-tenancy), ML platforms (feature stores, model registries, eval pipelines), and vector/embedding infrastructure.
• Pick your entry point. Inference serving is the highest-demand area. Training orchestration is the most operationally complex. ML platform is the closest to generic platform eng. Pick based on interest and available roles.
• Read 3 target JDs. Highlight the requirements you don’t meet. The gap list is shorter than you expect.

Milestone: you can name the 3–4 sub-specialties within AI infra, which one you’re targeting, and what your specific gaps are.

Phase 1: GPU systems and inference fundamentals (~4–6 weeks, ~8 hrs/week)

The core new content. This is where backend engineers need to build genuinely new mental models.

What to learn (in this order):

1. GPU memory hierarchy. HBM vs SRAM, memory bandwidth as the bottleneck for inference (not compute). Why a 70B model in fp16 needs 140GB and what tensor parallelism does about it. The NVIDIA H100 whitepaper is dense but the memory-bandwidth section is the key part.
2. LLM inference mechanics. Prefill vs decode, KV cache, why decode is memory-bandwidth-bound and prefill is compute-bound. Read the vLLM/PagedAttention paper — this is the single most important paper for AI infra engineers.
3. Continuous batching. How the scheduler admits and evicts requests, how paged attention enables variable-length batches, what preemption looks like. The Anyscale continuous batching post is the clearest explanation.
4. Model parallelism. Tensor parallelism (split layers across GPUs, all-reduce each step), pipeline parallelism (different layers on different GPUs), data parallelism (same model, different data). When to use each. The inter-node vs intra-node distinction (NVLink vs InfiniBand).
5. Quantization. fp16, fp8, int8, int4. The trade-off: smaller model → more requests per GPU, but lower quality. AWQ, GPTQ as quantization methods. Know what they do conceptually; don’t need to implement.
6. Serving frameworks. Read the architecture of vLLM, TensorRT-LLM, or SGLang. Pick one and read the scheduler code. This is worth a weekend.

What to skip:

• CUDA programming (unless targeting GPU kernel roles specifically).
• Training optimization (gradient accumulation, mixed precision training) — covered in Phase 2 if relevant.
• Model architecture design (you’re serving models, not building them).

Milestone: you can draw the inference serving path from request to token on a whiteboard, name the memory bottleneck, explain why continuous batching improves throughput 3–5x over static batching, and discuss tensor parallelism trade-offs.

Phase 2: ML workload understanding (~3–4 weeks, ~6 hrs/week)

Enough ML to understand what you’re serving and why the constraints exist. Not enough to build models.

What to learn:

1. Transformer architecture at operator level. Not the math — the compute graph. What operations happen in a forward pass, which are memory-bound, which are compute-bound, where the bottlenecks are.
2. Training workloads. What distributed training looks like (data parallelism, gradient synchronization, checkpointing). Why training is GPU-hours-expensive and what makes it fail (stragglers, communication overhead, OOM). DeepSpeed and FSDP as reference frameworks.
3. Embedding and retrieval workloads. How embedding models differ from generative models (fixed output, batchable, no KV cache). Why vector search is a different serving problem. How HNSW works conceptually.
4. Model lifecycle. How models move from training → evaluation → staging → production. Model registries, versioning, canary deploys for models. A/B testing at the model level.

What to skip:

• Loss functions, backpropagation math, hyperparameter tuning.
• Model selection (“which model is best for X”) — that’s the ML engineer’s job, not the infra engineer’s.
• Fine-tuning techniques beyond “it exists and produces a checkpoint.”

Milestone: you can explain the resource profile difference between serving a 7B model, a 70B model, and an embedding model, and you can describe the distributed training process (what’s communicated, what’s checkpointed, what fails).

Phase 3: Hands-on AI infra project (~4–6 weeks, ~10 hrs/week)

Build something. The project demonstrates you can do AI infra, not just talk about it.

Project ideas ranked by signal:

1. Build a multi-model inference gateway. A routing layer that accepts requests, selects the appropriate model (by tenant, by request type), manages rate limiting per tenant, and routes to the right vLLM/TGI backend. Deploy with Docker/K8s. Demonstrate prefix-cache-aware routing.
2. Build a GPU job scheduler. A simplified batch scheduler that accepts training jobs, bins them by GPU requirement, handles priority/preemption, and reports utilization. Demonstrates your distributed-systems skills applied to GPU resources.
3. Benchmark and optimize an inference setup. Take vLLM, deploy a model, benchmark throughput/latency under load, then optimize (tune batch sizes, try different quantization levels, add prefix caching). Write up the results with graphs.
4. Build a model deployment pipeline. CI/CD that takes a model checkpoint, runs eval, promotes to staging, runs canary, promotes to production. Demonstrates MLOps from the infra side.

What makes the project strong:

• It runs on real hardware (even a single GPU instance is fine).
• It demonstrates systems thinking (resource management, scheduling, routing, failure handling) — not ML knowledge.
• It has a write-up explaining the design decisions and trade-offs.

Milestone: a deployed project with a write-up that demonstrates AI-infra-specific systems work.

Milestones by phase

Resources by surface

GPU Systems & Inference

• vLLM PagedAttention paper — the primary reference. Read this first.
• calm.rocks: Designing an LLM Serving Platform — the system design version of Phase 1.
• Anyscale continuous batching — clearest end-to-end explanation of the scheduler loop.
• vLLM source code — reading the scheduler is worth a weekend.
• Skip: generic “intro to GPUs” content. You need the serving- specific view, not the CUDA programming view.

ML Workload Understanding

• Designing Machine Learning Systems — Chip Huyen. Chapters on serving and data pipelines most relevant.
• DeepSpeed documentation — reference for distributed training patterns.
• Skip: full ML courses. You need workload intuition, not model- building skill.

AI Infra Interview

• calm.rocks: AI/ML System Design — What’s Different — the three-layer framework.
• System Design Interview Vol. 2 — Alex Xu. General system design structure applies; supplement with GPU-specific knowledge.
• Skip: ML breadth prep (you’re not interviewing for ML knowledge; you’re interviewing for systems knowledge applied to ML workloads).

What to cut if time is short

If you have 6 weeks instead of 4–6 months:

• Phase 1: focus only on inference (skip training, skip embeddings). 2 weeks.
• Phase 2: skip entirely. Learn ML workload context from your project.
• Phase 3: build the benchmarking project (fastest to ship). 3 weeks.
• Interview prep: 1 week of system design mocks with AI-infra framing.

Never cut:

• The PagedAttention paper. It’s the foundational mental model.
• The hands-on project. Without it, you’re a backend engineer who read some papers, not a candidate who demonstrated AI-infra capability.
• Understanding the memory-bandwidth bottleneck. This is what makes you credible when talking about GPU systems.

Cut without guilt:

• Training orchestration depth (learn on the job if the role needs it).
• Quantization implementation details.
• More than one project.
• Distributed training math (gradient accumulation, etc.).

Common failure modes

Thinking you need to learn ML first. The most common detour. AI infra roles do not require you to build models. They require you to understand the resource profile of models — what they need from the system, what fails when the system doesn’t provide it. That’s a systems question, not an ML question. Don’t spend 3 months on ML fundamentals before touching infra.

Over-indexing on Kubernetes. Many AI infra roles use K8s. But K8s is not the differentiator — GPU-aware scheduling, memory management, and inference optimization are. Listing “deep K8s expertise” on your resume doesn’t distinguish you from any other platform engineer. The GPU-specific knowledge does.

Ignoring the reading-code path. The fastest way to understand AI infra is to read production inference code (vLLM, TGI, TensorRT-LLM). These are open-source, well-structured, and demonstrate every concept you need. An afternoon reading vLLM’s scheduler is worth a week of blog posts.

Targeting “AI Engineer” roles instead. “AI Engineer” at many companies means “application developer who calls LLM APIs.” That’s a different role with different skills (prompt engineering, RAG, agent frameworks). AI infrastructure engineer is a systems role. Make sure the JD matches — look for keywords like “serving,” “GPU,” “training platform,” “inference optimization,” not “prompt” or “agent.”

Waiting for a GPU before starting. You don’t need an H100 cluster to learn. Read code, read papers, draw architectures. When you’re ready for the project phase, a single A10G instance (~$1/hr on cloud) is enough to demonstrate the concepts.

After you pass

AI infra roles have steep learning curves in the first 90 days because the production systems are complex and the failure modes are GPU-specific. Your backend instincts (observability, capacity planning, incident response) will earn you credibility immediately. The new vocabulary (KV cache fragmentation, tensor-parallel communication overhead, quantization-induced quality regression) fills in as you operate the system.

Related

• Walkthrough: Designing an LLM Serving Platform — the full system design walkthrough for the inference side.
• Framework: What’s Durable vs What’s Hype — GPU memory constraints and inference mechanics are in the “durable substrate” quadrant.
• Plan: Transitioning to Machine Learning Engineer — the sibling plan if you decide applied ML is the better fit.
• Framework: AI/ML System Design — What’s Different — the serving layer (Layer 1) is your home turf.