Tutorials System Design

System Design

Learn system design from fundamentals to interview case studies — scalability, caching, databases, messaging, microservices, and reliability with chapter-wise practice tests.

Chapter 1

Introduction to System Design

What Is System Design?

System design is the process of defining architecture, components, modules, interfaces, and data for a software system to satisfy specified requirements. It sits between product requirements and implementation, translating user needs into scalable, reliable, and maintainable technical structures. In interviews and real projects, you explain how millions of users can read feeds, upload media, or place orders without collapsing a single server.

A useful mindset treats the system as a graph of services, storage, caches, queues, and clients connected by networks with latency, failure, and cost. You reason about read-heavy vs write-heavy workloads, hot keys, thundering herds, and regional outages. Good designs are boring in the right places: clear ownership boundaries, well-defined APIs, and observability everywhere.

Functional vs Non-Functional Requirements

Functional requirements describe what the system must do: users can post tweets, search products, or transfer money. Non-functional requirements describe how well it must do it: latency targets, availability, durability, security, compliance, and cost. NFRs drive architecture more than feature lists because they determine replication, caching, and partitioning strategies.

Always clarify scope before drawing boxes. Ask daily active users, read/write ratio, maximum payload size, retention period, and consistency expectations. A chat app needs low latency and ordering guarantees; a reporting dashboard tolerates minutes of staleness but must handle large scans efficiently.

  • Latency: p50, p95, and p99 targets for critical paths.
  • Availability: acceptable downtime per month (e.g., 99.9%).
  • Durability: can you lose data? For how long?
  • Scalability: growth over 1–3 years.

High-Level Building Blocks

Most large systems combine clients, load balancers, application servers, caches, primary databases, search indexes, object storage, message queues, and CDNs. Clients talk to edge layers first; static assets are served from CDNs close to users. Dynamic requests hit stateless app tiers that read through caches and fall back to databases.

Stateless application servers scale horizontally behind load balancers. Stateful data lives in specialized stores chosen for access patterns: relational databases for transactions, document stores for flexible schemas, key-value stores for sessions, column stores for analytics, and graph databases for relationships.

The Design Process

Start with requirements and back-of-the-envelope estimates. Sketch a simple diagram with main components and data flow. Identify bottlenecks and scale them independently. Discuss failure modes and monitoring. Iterate by deepening the riskiest component—often the database or hot partition.

In interviews, narrate your thinking aloud. State assumptions, propose a baseline, then improve it. Interviewers reward clarity, trade-off analysis, and pragmatic choices over exotic technologies.

Common Pitfalls for Beginners

Junior designs often over-engineer microservices on day one or under-estimate data growth. Another mistake is ignoring the read path: a write-optimized schema may collapse when feeds or dashboards scan huge tables. Always ask who reads the data, how often, and with what filters.

Another pitfall is skipping failure analysis. Single-region deployments, missing backups, and no rate limits are acceptable only when explicitly scoped. Production traffic is spiky; exams, sales, and viral posts appear without warning.

  • Do not cache without a defined invalidation story.
  • Do not shard before exhausting simpler scale options.
  • Do not hide complexity inside synchronous call chains.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 2

Requirements and Capacity Estimation

Gathering Requirements

Requirements workshops align product, design, and engineering on scope and success metrics. Capture user personas, core flows, edge cases, and regulatory constraints. For a URL shortener, functional needs include shorten, redirect, and optional analytics; non-functional needs include low redirect latency and global availability.

Ambiguous requirements cause rework. Document invariants: unique short codes, idempotent creates, maximum URL length, and TTL for inactive links. Ask whether custom aliases, bot protection, or private links are in scope.

Back-of-the-Envelope Math

Estimation prevents designs that are orders of magnitude wrong. Convert DAU to QPS using peak multipliers. Example: 100M DAU, each user 10 redirects/day → 1B redirects/day ≈ 12k average QPS; peak might be 3–5× higher.

Storage estimates multiply object size by volume and retention. 1B new URLs/month at 500 bytes metadata each ≈ 500 GB/month before indexes and replicas. Bandwidth follows payload size times QPS. These numbers guide sharding, cache size, and CDN budgets.

  • 1 day = 86,400 seconds; use for QPS.
  • Peak traffic often 2–5× average.
  • Add 30–50% headroom for growth and failures.

Read vs Write Ratio

Read-heavy systems benefit from aggressive caching, read replicas, and CDNs. Write-heavy systems stress ingestion pipelines, WAL throughput, and partition balance. Mixed workloads need separate scaling paths for read and write paths, sometimes CQRS to split models.

Estimate separately: writes drive storage growth and consistency complexity; reads drive cache footprint and replica count.

SLIs, SLOs, and Error Budgets

Service Level Indicators measure behavior (latency, error rate). Service Level Objectives set targets (99% of redirects < 100ms). Error budgets connect reliability to feature velocity: if you burn budget on outages, freeze risky launches and invest in stability.

Design choices should map to SLOs. Multi-region active-active costs more but improves availability SLOs for global users.

Worked Example: Estimating a Photo App

Suppose 10M DAU upload 2 photos/day and view 50 photos/day. Writes: 20M photos/day; reads: 500M views/day. Average photo metadata 1 KB plus 2 MB file in object storage. Daily storage growth ~40 TB of blobs plus 20 GB metadata—dominated by objects, so use S3-style storage with lifecycle policies.

Read QPS ≈ 500M / 86400 ≈ 5.8k average, higher at peak. Thumbnail and feed endpoints should hit CDN and cache; uploads go through API to object store and async workers for resizing.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 3

Scalability and Performance

Vertical vs Horizontal Scaling

Vertical scaling increases CPU, RAM, or disk on one machine. It is simple but hits hardware limits and creates single points of failure. Horizontal scaling adds nodes, requiring load distribution and often application redesign for statelessness.

Most web tiers scale horizontally; databases scale vertically first, then replicate and shard. Choose horizontal scaling when traffic growth is unpredictable or multi-region resilience is required.

Performance Fundamentals

Latency is the time to complete one operation; throughput is operations per second. Optimizing throughput without watching tail latency can make user experience worse. Profile critical paths, eliminate N+1 queries, batch network calls, and prefer asynchronous work for non-critical tasks.

The utilization-latency curve is non-linear: past ~70% utilization, queues grow and p99 latency spikes. Autoscaling should trigger before saturation, using predictive signals where possible.

Bottleneck Analysis

Find the slowest hop: DNS, TLS, app CPU, lock contention, disk I/O, or remote service. Use tracing, metrics, and load tests. Common fixes include connection pooling, indexing, caching hot keys, and splitting monoliths when teams block each other.

Amdahl's Law reminds us parallelization only helps the parallelizable portion. Optimize the serial bottleneck first.

  • Measure before caching blindly.
  • Watch connection pool exhaustion.
  • Monitor GC pauses on JVM languages.
  • Track saturation per dependency.

Auto Scaling and Elasticity

Autoscaling policies use CPU, request rate, queue depth, or custom metrics. Cooldown periods prevent flapping. Pre-warm instances before known events (sales, exams). Serverless offers instant elasticity with cold-start trade-offs.

Latency Numbers Every Designer Should Know

Orders of magnitude matter in interviews. L1 cache references are nanoseconds; SSD random reads are microseconds to low milliseconds; cross-region RPC is tens to hundreds of milliseconds. Serialization and chatty APIs dominate at scale more often than CPU math.

Design to keep user-critical paths short: fewer hops, parallel independent fetches where possible, and push heavy work offline. Paginate aggressively and stream large payloads.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 4

Load Balancing

Role of Load Balancers

Load balancers distribute traffic across healthy backends, terminate TLS, enforce routing rules, and provide a stable endpoint while instances churn. They sit between clients and app servers, and also between app tiers and databases in some setups.

Layer 4 balancers route by IP and port with minimal inspection—fast and simple. Layer 7 balancers understand HTTP, enabling path-based routing, header rewrites, and sticky sessions.

Load Balancing Algorithms

Round robin cycles through servers; weighted round robin favors stronger machines. Least connections sends traffic to the least busy server—good for long-lived connections. IP hash can provide soft stickiness without cookies.

Consistent hashing minimizes remapping when nodes are added or removed, important for distributed caches and sharded systems.

  • Round robin: even distribution for homogeneous work.
  • Least connections: better for variable request duration.
  • Random with two choices: cheap approximate balance.

Health Checks and Failover

Active health checks probe endpoints; passive checks use real traffic errors. Aggressive timeouts remove flaky nodes quickly but risk false positives during deploys. Connection draining completes in-flight requests before removal.

Global load balancing uses DNS or anycast to route users to the nearest healthy region, combining with regional L7 balancers.

Sticky Sessions and State

Session stickiness routes the same client to the same server, simplifying server-side session storage but complicating scale-in and failures. Prefer external session stores (Redis) with stateless apps when possible.

DNS and Global Traffic Management

DNS TTL affects how quickly traffic shifts during failover. Low TTL enables faster cutover but increases query load. GeoDNS returns different IPs per region. Anycast advertises the same IP from multiple POPs, letting BGP route to nearest healthy edge.

Combine global DNS with health-checked regional stacks. Clients should retry alternate regions only when designed for it; otherwise users may see confusing partial failures.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 5

Caching and CDN

Why Caching Matters

Caches store copies of expensive-to-compute or expensive-to-fetch data closer to readers. A cache hit avoids database round trips and reduces cost. Effective caching requires understanding access patterns: temporal locality (recent data reused) and spatial locality (related data accessed together).

Caches trade freshness for speed. Define TTL policies, invalidation rules, and what happens on miss (singleflight to prevent stampedes).

Cache Placement

Client caches (browser), CDN edge, reverse proxy (Varnish/Nginx), application in-memory cache, distributed cache (Redis/Memcached), and database buffer pools form a hierarchy. Each layer has different hit ratio, latency, and invalidation complexity.

Cache-aside: app reads cache, on miss loads DB and populates cache. Write-through: writes go to cache and DB together. Write-back: writes go to cache first, async flush to DB—fast but riskier.

  • CDN: static assets and cacheable API responses at edge.
  • Redis: sessions, rate limits, hot objects.
  • Application cache: computed views per instance—watch consistency.

Cache Invalidation

Invalidation is famously hard. Strategies include TTL expiry, event-driven purge, versioned keys, and materialized views rebuilt on schedule. For highly dynamic data, shorten TTL or bypass cache on critical reads.

Thundering herd: many clients miss simultaneously. Mitigate with request coalescing, jittered TTL, and prewarming after deploys.

Content Delivery Networks

CDNs replicate static and streaming content globally, reducing latency and origin load. Use cache-control headers, signed URLs for private content, and origin shielding. Dynamic site acceleration can cache HTML fragments when safe.

Designing Cache Keys and Hot Keys

Cache keys should include tenant, locale, or version when those dimensions change responses. Avoid unbounded key cardinality that explodes memory. Hot keys on a single Redis node can be mitigated by local in-process caching, read replicas of cache, or splitting the key into logical shards.

Monitor hit ratio, evictions, and memory fragmentation. A cache that thrashes wastes money and adds latency on misses.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 6

Databases and Data Modeling

SQL vs NoSQL

Relational databases excel at structured data, joins, and ACID transactions. NoSQL families optimize for scale-out and flexible schemas: document (MongoDB), key-value (Dynamo-style), wide-column (Cassandra), graph (Neo4j). Choose based on access patterns, not hype.

Many systems use polyglot persistence: PostgreSQL for orders, Elasticsearch for search, S3 for blobs, Redis for cache.

Indexing and Query Planning

Indexes accelerate reads but slow writes and consume storage. Composite indexes must match query predicates order. Covering indexes avoid table lookups. Explain plans reveal full table scans to fix before production.

Avoid SELECT * in hot paths; paginate with keyset pagination instead of large OFFSET.

  • B-tree indexes: range queries and sorting.
  • Hash indexes: equality lookups.
  • Full-text indexes: search features.

Normalization vs Denormalization

Normalization reduces redundancy and anomalies; denormalization duplicates data for read performance. In distributed systems, denormalization is common because joins across shards are expensive. Eventual consistency between duplicates must be managed.

Transactions and Isolation

ACID transactions protect invariants within a database. Distributed transactions (2PC) are costly; prefer sagas, outbox pattern, and idempotent consumers for cross-service consistency.

Choosing the Right Data Store

Match the store to access patterns, not logos. OLTP row stores for accounts and payments; search engines for full-text; OLAP warehouses for BI; time-series DB for metrics; blob storage for media. Migrating later is expensive—get boundaries right early.

Schema migrations in live systems need expand/contract patterns: add nullable columns, dual-write, backfill, then remove old fields. Plan zero-downtime deploy compatibility.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 7

Replication, Partitioning, and Sharding

Database Replication

Replication copies data for read scale and failover. Leader-follower: writes go to primary, replicas serve reads with replication lag. Multi-leader increases write availability but complicates conflicts. Leaderless (quorum) systems trade tunable consistency.

Monitor replication lag; stale reads break assumptions unless clients request strong consistency explicitly.

Partitioning Strategies

Partitioning splits data by range, hash, or directory. Range partitions enable range queries but risk hot spots (latest time bucket). Hash partitioning spreads load but complicates range scans. Directory service maps keys to shards flexibly.

  • Hash mod N: simple but painful resharding.
  • Consistent hashing: smoother cluster changes.
  • Geo partitioning: data residency compliance.

Sharding in Practice

Sharding partitions data across independent database instances. Choose shard key carefully—user_id often works; avoid monotonic keys that hotspot one shard. Cross-shard queries are expensive; design APIs to be shard-local when possible.

Resharding requires dual writes, backfill, and cutover plans. Virtual shards (many logical shards, fewer physical nodes) ease migration.

CAP and PACELC

CAP: during a network partition, choose consistency or availability. PACELC extends: else (normal operation), choose latency vs consistency. Document which failures your product tolerates.

Replication Lag in User Experience

Users notice stale followers counts or delayed notifications. Product copy and UX can mask eventual consistency, or code can read-your-writes by routing to primary after mutation. Analytics pipelines tolerate minutes of lag; fraud checks may not.

Failover drills reveal whether RPO/RTO targets are real. Automated promotion without data loss needs synchronous replication or consensus—at a latency cost.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 8

Message Queues and Async Processing

Asynchronous Architecture

Queues decouple producers and consumers, absorbing traffic spikes and enabling parallel processing. A web app can acknowledge user actions quickly while workers index, send email, or generate thumbnails in the background.

Message systems include point-to-point queues (SQS) and publish-subscribe topics (Kafka, SNS). Kafka adds durable logs for replay and stream processing.

Delivery Semantics

At-most-once may lose messages but never duplicates—ok for metrics. At-least-once retries until ack; consumers must be idempotent. Exactly-once is hard end-to-end; use idempotent keys and transactional outbox patterns.

  • Visibility timeout: message hidden while processing.
  • Dead letter queue: poison messages isolated.
  • Ordering: single partition preserves order in Kafka.

Event-Driven Design

Events announce facts (OrderPlaced) rather than commands. Downstream services react independently, enabling extensibility. Schema evolution and contract testing prevent breaking consumers.

CQRS separates write model from optimized read models updated by events.

Backpressure and Flow Control

When consumers lag, queues grow unbounded. Apply rate limits, scale consumers, shed load, or drop low-priority work. Monitor age-of-oldest-message alarms.

Stream Processing Overview

Beyond simple queues, log-based systems enable stream processors to compute aggregates, fraud scores, or recommendations in near real time. Windowed joins and exactly-once semantics require careful framework configuration.

Choose batch vs stream based on latency SLO and cost. Nightly ETL is cheaper; sub-minute insights need streaming investment.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 9

Microservices and API Design

Monolith vs Microservices

Monoliths are simpler to deploy and debug early. Microservices split by business capability, enabling independent scaling and team ownership at the cost of distributed complexity: networking, versioning, observability, and data consistency.

Extract services when boundaries are clear and team coordination pain exceeds operational overhead. Start modular monolith if unsure.

API Styles

REST uses resources and HTTP verbs—ubiquitous and cache-friendly. gRPC uses protobuf over HTTP/2—efficient for internal RPC. GraphQL lets clients specify fields—reduces over-fetching but needs query cost limits.

Version APIs explicitly (/v1), deprecate gracefully, and document error models and rate limits.

  • Idempotency-Key header for safe retries on POST.
  • Pagination cursors for stable lists.
  • Correlation IDs across service calls.

Service Mesh and Discovery

Service discovery (Consul, Kubernetes DNS) maps logical names to instances. Service meshes (Istio) add mTLS, retries, and traffic splitting without baking into each app.

Fault Tolerance Patterns

Timeouts, retries with exponential backoff and jitter, circuit breakers, bulkheads, and fallbacks prevent cascading failures. Chaos engineering validates resilience assumptions.

API Gateways and BFF Pattern

API gateways centralize authentication, rate limiting, and routing. Backend-for-frontend services tailor payloads per client (mobile vs web), reducing chattiness and over-fetching. Avoid turning the gateway into a second monolith with all business logic.

Document ownership: who runs the gateway, who owns schema compatibility, and how breaking changes roll out.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Chapter Practice Test

10 questions — answer all and submit to see your score.

Chapter 10

Interview Framework and Case Studies

Structured Interview Approach

Use a repeatable framework: clarify requirements, estimate scale, propose high-level design, deep dive on critical components, address bottlenecks and failures, and summarize trade-offs. Spend five to ten minutes on requirements before drawing.

Communicate continuously. When stuck, state options and pick one with justification.

Designing a URL Shortener

Requirements: shorten URLs, redirect, analytics optional. Estimate QPS and storage. API: POST /urls, GET /{code}. Store mapping in database with unique index on code; use base62 encoding of incremental IDs or random tokens with collision check. Cache hot redirects in Redis. Scale reads with replicas and CDN for 301 redirects.

Designing a News Feed

Fan-out on write pushes posts to follower feeds at publish time—fast reads, heavy writes for celebrities. Fan-out on read composes feed at request time—cheaper writes, slower reads. Hybrid: fan-out for normal users, merge celebrity posts on read. Use ranking, pagination cursors, and media in object storage.

  • Identify read vs write heavy paths.
  • Plan hot user mitigation.
  • Consider eventual consistency for likes counts.

Designing Chat or Notifications

WebSockets or long polling for real-time delivery; message store partitioned by conversation_id. Presence and typing indicators need low-latency pub/sub. Push notifications use mobile gateway providers with retry queues.

Final Tips

Practice drawing diagrams cleanly. Know numbers: RAM vs SSD vs network latency orders of magnitude. Tie every box to a metric. End with what you would monitor and how you would roll out safely.

Mock Interview Drill

Timebox 45 minutes: 5 requirements, 10 estimation, 15 high-level design, 10 deep dive, 5 failure modes. Record yourself. Note where you hand-wave—those are study gaps.

Compare your design to how real products evolved (Twitter fan-out, Uber dispatch, Netflix CDN). Reality adds constraints politics and legacy; interviews simplify but expect follow-up why questions.

Design Trade-offs in Practice

Every production system is a bundle of trade-offs rather than a perfect architecture diagram. Teams choose synchronous APIs when consistency and simplicity matter, and asynchronous pipelines when throughput and decoupling dominate. The same company may use strong consistency for billing and eventual consistency for analytics because the business cost of failure differs. Interview answers become stronger when you name the trade-off explicitly and justify it with user impact, operational cost, and recovery time.

Operational maturity also changes the right answer. A startup may accept a single database and manual failover because engineering time is scarce, while a mature product invests in multi-region replication and automated runbooks. Document assumptions about traffic shape, data growth, and compliance early so design decisions remain valid six months later.

  • Latency vs throughput: optimize the metric users feel first.
  • Consistency vs availability: use CAP reasoning with real failure scenarios.
  • Build vs buy: managed services reduce toil but add vendor dependency.
  • Cost vs resilience: duplicate components only where outages hurt revenue.

Reliability Checklist

Reliable systems define SLOs, measure them continuously, and alert on burn rate rather than single spikes. Retries must be idempotent, timeouts must be explicit, and cascading failures must be blocked with bulkheads and circuit breakers. Backups and disaster recovery drills are part of architecture, not an afterthought.

Security belongs in system design: authenticate callers, authorize actions, encrypt data in transit and at rest, and audit sensitive operations. Rate limiting protects shared components from abuse and accidental hot keys.

  • Set timeouts on every outbound network call.
  • Use health checks that reflect real dependencies.
  • Prefer graceful degradation over total outage.
  • Run game days to validate failover assumptions.

Chapter Summary

System design is iterative. Start with requirements, sketch a high-level diagram, estimate capacity, identify bottlenecks, add scaling layers, and revisit consistency and failure modes. The goal is not memorizing buzzwords but explaining how data flows, where state lives, and what breaks first under load.

Practice explaining your design to a teammate who will challenge assumptions. The best architects combine quantitative estimates with operational empathy: deploys, on-call pain, and customer trust.

Monitoring and Observability

Metrics expose RED/USE signals: rate, errors, duration for services; utilization, saturation, errors for resources. Distributed tracing follows requests across hops. Structured logs with correlation IDs tie user reports to failing spans. Dashboards without alerts are vanity; alerts without runbooks waste on-call energy.

Define golden signals per service: latency, traffic, errors, saturation. SLO dashboards show budget remaining. Post-incident reviews update architecture docs and action items—not blame slides.

  • Instrument the critical user journey end to end.
  • Alert on symptom-based SLO burn, not every CPU blip.
  • Store logs with retention matched to compliance needs.

Cost and Capacity Planning

Cloud bills scale with egress, stored bytes, and over-provisioned idle instances. Reserved capacity and autoscaling save money when traffic is predictable. Archive cold data to cheaper tiers. Right-size databases before multiplying replicas.

Cost-aware design chooses managed services when staff time is more expensive than markup. Batch and compress data movement across regions.

Security and Compliance in Architecture

Threat modeling STRIDE identifies spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Segment networks, least-privilege IAM, secrets rotation, and audit trails are baseline—not optional extras.

Data residency and GDPR/CCPA may force regional storage and deletion workflows. Bake retention and export APIs into design early.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Rollouts should be progressive: canary, region-by-region, feature flags. Instant global deploys maximize blast radius of subtle bugs.

On-call health improves when architectures expose clear health endpoints per dependency and automate rollbacks when error budgets burn too fast during a release.

When two designs both work, prefer the one with fewer moving parts and clearer ownership. Operational simplicity reduces mean time to recovery and makes onboarding faster for new engineers joining the team.

Capacity planning should include failure capacity: if one availability zone disappears, remaining zones must absorb traffic without violating SLOs. Load tests should simulate dependency timeouts, not only happy paths.

Data models outlive services. Investing time in entity boundaries, id formats, and migration strategy pays off across years of feature work and acquisitions.

Every external dependency needs a fallback story: degrade features, serve cached snapshots, or queue work for later. Silent hangs hurt NPS more than explicit error messages with retry guidance.

Platform teams succeed when they offer paved roads—templates, CI/CD, observability baselines—so product squads do not reinvent shards and caches per feature.

Document ADRs (Architecture Decision Records) when you choose Kafka over RabbitMQ or SQL over NoSQL. Future you will forget the constraints that justified the choice.

Geographic expansion introduces replication lag, legal boundaries, and clock skew. Design APIs idempotently and use version vectors or timestamps consciously.

Performance testing in staging must use anonymized production-shaped data volumes. Empty tables lie about index selectivity and planner behavior.

Chapter Practice Test

10 questions — answer all and submit to see your score.