4 October 202615 min read

Avoid Premature Sharding: Kubernetes Patterns for Web App Scalability

Two checkout lanes in a 1980s supermarket, the second lane's amber lamp just switching on as it opens.

Scalable web apps deliver consistent latency and availability under growing load by combining the right architecture with autoscaling, caching, data partitioning, and SLO-driven testing. The core levers are architecture choice, database scaling patterns, caching and CDN design, autoscaling, and continuous testing and monitoring. If you need one immediate action, find your most glaring bottleneck (usually the database or an uncached hot path) and fix that first before investing in autoscaling or infrastructure changes.

TL;DR:

Focus first on fixing the most critical bottleneck, usually the database or an uncached hot path, before investing in autoscaling or infrastructure overhauls.

Horizontal scaling requires stateless, idempotent, and loosely coupled services, while vertical scaling is limited by instance type caps and stateful service constraints.

Prioritize query tuning and adding read replicas to improve database capacity before considering complex sharding, which introduces significant complexity.

Use open-loop load testing at high traffic rates to accurately measure system limits, focusing on percentile latency and graceful degradation under overload.

Implement strict rate limiting, properly configured cache security, and careful cache invalidation, as scaling amplifies exposure and security risks.

Types of scaling and the principles that make them work

Vertical scaling means giving a single server more CPU, memory, or disk. It is simple and often the fastest fix, but every cloud instance type has a ceiling, and stateful services that depend on local disk or in-memory session data hit that ceiling hard. Horizontal scaling means adding more machines or instances behind a load balancer. It has no practical ceiling, but it demands that your application tier can run many copies of itself without stepping on its own state. Diagonal scaling blends the two: you scale up each node a bit and scale out the number of nodes, which is how most production systems actually grow over time.

Choosing between them depends on your constraints. A single-tenant analytics job that needs a lot of memory for one batch run is a good vertical-scaling candidate. A public API handling unpredictable traffic spikes needs horizontal scaling. Managed cloud services often cap vertical scaling at a specific instance tier, which pushes you toward horizontal scaling sooner than you might expect.

Horizontal scaling only works if your services follow a few design principles:

Statelessness: no request should depend on data stored only on the instance that handled the previous request.
Idempotency: retried requests (common under load balancing and autoscaling) must not cause duplicate side effects.
Bounded contexts: each service owns a clear slice of data and logic, which keeps horizontal replicas independent of each other.

Cloud architecture guidance from Google Cloud’s reliability framework recommends re-architecting toward horizontal scalability and stateless design specifically because it removes the single-instance ceiling that vertical scaling eventually hits.

Scaling the data layer without a rewrite

The database is where most scaling plans break down, mainly because teams reach for sharding before trying cheaper fixes. Start by identifying slow queries with your database’s query planner or a tool like pg_stat_statements, then add indexes that match your actual query patterns rather than guessing. A surprising number of “scaling problems” disappear once the right composite index exists.

Once indexing and query tuning are exhausted, read replicas are the next lever. They copy data asynchronously to one or more secondary nodes, and you route read-heavy traffic (dashboards, reporting, search) to the replicas while writes stay on the primary. The catch is replication lag: replicas are not instantly consistent, so any feature that reads its own writes immediately (a user posting a comment and expecting to see it) needs to either read from the primary or tolerate a short delay.

Sharding, by contrast, splits data horizontally across multiple independent databases, each holding a subset of rows (by customer ID, region, or another key). It solves write-throughput limits that replication cannot touch, but it adds real complexity: cross-shard queries, rebalancing, and migration logic. Reach for sharding only after replication and indexing stop being enough, not as a first move.

Practical steps that apply regardless of scale:

Profile before optimizing: use slow-query logs to find the actual bottleneck instead of assuming.
Add read replicas for reporting and search traffic before considering sharding.
Monitor replication lag continuously, since a lagging replica silently serving stale data is worse than no replica at all.
Use a connection pooler like pgbouncer to prevent connection exhaustion under load, since most databases have hard limits on concurrent connections and a chatty application can hit that ceiling long before CPU or memory does.

Pro Tip: Before adding a replica or shard, check whether a connection pooler alone fixes your symptoms: many “database is overloaded” incidents are actually “too many idle connections” incidents.

Managed database services (Amazon RDS, Cloud SQL, PlanetScale) trade some control for predictable scaling, automated backups, and less operational burden. They make sense when your team is small and your growth is predictable. Self-hosting makes sense when you need a specific extension, tight cost control at very high scale, or configuration a managed service does not expose. The migration path between the two gets harder the longer you wait, so decide early rather than mid-crisis.

Caching, CDNs, and keeping cache design secure

Caching is the highest-leverage lever on this list because it removes work entirely rather than distributing it. Static assets (JavaScript bundles, images, fonts) belong at the CDN edge with long cache lifetimes. Dynamic, user-specific responses need shorter-lived caches or none at all, and anything containing authorization-sensitive data should default to Cache-Control: private or no-store unless you have explicitly verified it is safe to share.

Cache-key design is where most cache-related security problems start. The OWASP Web Cache Security Cheat Sheet treats caching as part of the authorization model: if a request input (a header, a cookie, a query parameter) can change the response but is not included in the cache key, an attacker can poison the cache or one user can end up served another user’s cached response. Every input that affects the response belongs in the key, a Vary header, or the route should not be cached at all.

Cache static assets at the edge with long max-age and versioned URLs.
Keep dynamic, personalized responses out of shared caches unless the key fully captures what makes them personal.
Use Vary headers deliberately, not as a default, since an overly broad Vary can defeat caching entirely.
Purge by key pattern, not by guesswork, and test invalidation paths in a staging environment that mirrors production traffic.

Treating cache configuration as a security boundary, not just a performance knob, is the practice OWASP’s cheat sheet centers its guidance on, since cache misconfiguration is a recurring cause of cross-tenant data exposure.

CDN selection is also a trust decision: you are extending your security perimeter to a third party. Subresource Integrity (SRI) lets browsers verify that a fetched script or stylesheet matches an expected hash, and tools in the Sigstore ecosystem extend that verification to signed software artifacts, which limits the damage if a CDN or dependency is compromised.

Architecture foundations that make scaling possible

No amount of autoscaling fixes an architecture that was not built to run in parallel. Stateless front-end services, where any instance can handle any request, are what make horizontal scaling and load balancing work at all. The moment a service stores session state in local memory or writes to local disk, you have created a single point that cannot be freely replicated, and you will eventually need sticky sessions or a shared session store (Redis is the common choice) to route around it.

Service boundaries matter just as much as statelessness. A modular monolith, a single deployable application internally organized into well-separated modules, can scale a long way if each module is clean and the whole thing is stateless. It is simpler to operate, test, and deploy than a microservices architecture, and it is the right starting point for most teams. Microservices earn their complexity when different parts of the system genuinely need independent scaling, deployment, or ownership, not just because the architecture diagram looks more sophisticated. The practical migration path is to extract a service only when a specific module has outgrown the monolith’s scaling or team-ownership model, not all at once.

Event-driven patterns decouple the parts of your system that run at different speeds. A write to a queue (RabbitMQ, Amazon SQS) or a stream (Kafka, Kinesis) lets a user-facing request return quickly while background workers handle the slower processing asynchronously. This matters most for write-heavy workloads: an e-commerce checkout, for instance, can accept the order and queue inventory updates, email confirmation, and analytics events rather than making the customer wait for all of them to complete synchronously.

Keep front-end and API services stateless so any instance can serve any request.
Start with a modular monolith and extract services only when a module has outgrown it.
Use queues or streams to decouple slow background work from the request path that users wait on.

Combined, these choices determine whether autoscaling actually helps. Adding more instances of a stateful, tightly coupled service often just adds more places for bugs to hide, not more capacity.

Autoscaling in practice

Autoscaling comes in three modes, and picking the wrong one is a common failure point. Metric-based autoscaling reacts to live signals like requests per second or queue depth. Scheduled autoscaling anticipates known patterns, a retailer’s traffic jump every evening, for example. Predictive autoscaling forecasts demand ahead of time and is most valuable when provisioning is slow, such as spinning up GPU nodes, where reacting after the spike arrives is already too late.

On Kubernetes, four building blocks cover most cases:

Horizontal Pod Autoscaler (HPA) adds or removes pod replicas based on CPU, memory, or custom metrics like requests per second.
Vertical Pod Autoscaler (VPA) adjusts the CPU and memory requests of existing pods so they are sized correctly without manual tuning.
Cluster Autoscaler or Karpenter adds or removes nodes so there is enough underlying capacity for the pods HPA wants to run; Karpenter provisions nodes just-in-time rather than from fixed node-group templates.
KEDA extends autoscaling to event-driven triggers, scaling pods based on queue length or stream lag rather than CPU alone.

A CNCF overview of autoscaling trade-offs describes KEDA and Karpenter as complementary: KEDA decides how many pods are needed based on custom triggers, and Karpenter makes sure nodes exist to run them, together balancing cost, reliability, and performance better than CPU-based scaling alone.

Predictive autoscaling earns its complexity specifically for slow-provisioning workloads. A CNCF case study on GPU autoscaling describes a controller combining a short-term forecasting model with burst detection and graduated scaling, scaling out in waves rather than all at once, that reached 85% accuracy within plus or minus 10% at ten minutes ahead and prevented cascading failures in testing. Graduated scaling matters beyond GPUs too: scaling out hundreds of pods simultaneously can overwhelm a scheduler and cause the exact instability autoscaling was supposed to prevent.

Pro Tip: Test autoscaling behavior in a staging environment with synthetic load before trusting it in production, and set a hard maximum replica count so a runaway metric cannot scale you into a surprise bill.

Load testing that produces numbers you can trust

Closed-loop load tests, where each virtual user waits for a response before sending the next request, understate how real traffic behaves because real users do not politely wait their turn. Open-loop tests, which send requests at a fixed rate regardless of response time, more accurately reveal how a system behaves as it approaches its limits. Google Cloud’s load-testing guidance recommends open-loop patterns and specifies the metrics worth recording: throughput, 99th-percentile latency, concurrency, and error rate, rather than relying on a single average-response-time number that hides the worst experiences.

Use open-loop tools like k6 or Nighthawk to simulate realistic, non-waiting traffic.
Record 99th-percentile latency, not just the average, since averages hide the slow requests that drive complaints.
Run deliberate overload tests to see how the system fails, and build load-shedding (rejecting excess requests cleanly) so it degrades gracefully instead of falling over entirely.
Feed the results into autoscaler thresholds and SLOs directly, since a tested capacity number is far more reliable than an estimate.

Monitoring, SLOs, and turning telemetry into action

Service Level Objectives translate “the system should be fast and reliable” into numbers you can alert on, like “99% of requests complete in under 300 milliseconds.” An error budget, the allowed amount of SLO violation before you pause feature work for reliability work, turns that number into a decision-making tool rather than a dashboard nobody checks.

Track percentile latencies (p50, p95, p99), request rate, error rate, and queue depth as your core signals.
Use Prometheus for metrics collection and distributed tracing to connect a slow user-facing request to the specific service or query causing it.
Automate the obvious remediations, like restarting a stuck pod or scaling out when queue depth crosses a known threshold, so humans handle only the incidents that actually need judgment.

Scaling storage: OLTP, OLAP, and event-driven data

Transactional and analytical workloads compete for the same database resources if you let them share one system. Keep your OLTP database (the one handling live user transactions) focused on fast, small reads and writes, and move reporting, dashboards, and analytics to a separate OLAP store or data warehouse that is optimized for scanning large volumes of historical data. Running a heavy analytics query against your production transactional database is a common and entirely avoidable cause of latency spikes.

Large assets, images, videos, uploaded files, belong in object storage (Amazon S3, Cloud Storage) behind a CDN, not in your primary database. Serve them through immutable, versioned URLs (a filename that includes a content hash) so the CDN can cache them indefinitely without ever needing to check for changes.

Separate OLTP and OLAP workloads into different stores so reporting queries cannot starve transactional traffic.
Store large binary assets in object storage, not the database, and serve them via CDN.
Use versioned, immutable URLs for cacheable assets so cache invalidation becomes unnecessary.

For write-heavy systems, event sourcing and CQRS (separating the write model from the read model) let you scale writes and reads independently, with background processors building optimized read views from a stream of events rather than querying the live transactional store directly.

A cost and trade-off checklist before you invest

Not every lever is worth pulling for every team. Caching and query fixes are quick wins with high return. Sharding, multi-region deployment, and predictive autoscaling are long-term investments that make sense only once simpler fixes are exhausted and growth is sustained.

Fix caching and slow queries first; they are cheap and often remove most of the pain.
Add read replicas before sharding; sharding is a last resort, not a starting point.
Invest in predictive autoscaling only for slow-provisioning, predictable workloads like GPU-backed services.
Run an SLO and load-testing pass before any infrastructure change, so you know what you are actually fixing.

Action	Typical cost	When it pays off
Add database indexes	Low	Immediately, for most query-heavy systems
Add a CDN and static caching	Low	As soon as you serve static assets to real users
Add read replicas	Medium	When read traffic outgrows a single primary
Implement sharding	High	Only after replication is exhausted
Predictive autoscaling	High	Slow-provisioning workloads with predictable spikes

What building these systems actually looks like in practice

Scalability work rarely fails because a team picked the wrong tool. It fails because requirements shift mid-project, ownership changes hands, or the person who understood the original architecture leaves before the bottleneck is fixed. A senior-led model keeps the same engineers involved from the first consultation through handover, specifically to avoid the mid-project rewrites and dependency traps that derail scaling work elsewhere.

Ampersand’s case studies include projects like running a multi-site platform for a Swiss political party from a single system, where the scaling challenge was less about raw traffic and more about operating many near-identical sites reliably from shared infrastructure, a pattern that rewards the same statelessness and clean service-boundary principles covered above. The practical next step for a team evaluating its own scaling readiness is an audit or a scoped pilot before committing to a larger rebuild, rather than guessing at the right architecture up front.

Security considerations that scaling introduces

Scaling a system up also scales its exposure. Every new instance, cache layer, and public endpoint is another surface an attacker can probe, and the defenses that mattered at low traffic (a single firewall rule, a manually reviewed request) stop working once traffic is distributed across dozens of autoscaled instances.

Rate limiting is the first line of defense, and it needs to happen before a request reaches your expensive application logic, ideally at the load balancer or API gateway, not deep inside a service that has already spent CPU cycles parsing the request. Per-user and per-IP limits catch different problems: per-user limits stop abuse from a single compromised account, while per-IP limits catch broader scraping or brute-force attempts, though they can also catch legitimate users behind shared network address translation.

DDoS protection at scale generally relies on a CDN or dedicated service absorbing volumetric attacks before they reach origin servers, since trying to autoscale your way out of a distributed denial-of-service attack is both slow and expensive compared to filtering at the edge.

Caching and scaling also interact with security in ways that are easy to miss: a cache that is not keyed correctly can serve one user’s rate-limited error page to another, or worse, serve a personalized, authenticated response to an unauthenticated visitor, which is exactly the cache-poisoning risk the OWASP cheat sheet addresses. As you distribute a system across more CDN edges, more autoscaled instances, and more cache layers, treat every one of those additions as a new point that needs its own access controls, not just more capacity.

How Ampersand helps teams scale with less guesswork

Scaling a web application well means getting architecture, data, caching, and operations right at the same time, which is hard to do with a shifting cast of contractors. Ampersand keeps the same senior engineers on a project from first consultation to handover, whether that means building a new web application or platform, setting up system integrations, or providing ongoing technical support through a monthly retainer. If your system needs a capacity assessment or a scoped pilot before a larger investment, check Ampersand’s engagement models and pricing and get in touch to scope the work.

FAQ

What is scalability in a web application?

Scalability is a system’s ability to handle increasing load, more users, more data, more requests, without a proportional loss in performance or reliability. It is usually achieved through a combination of horizontal scaling, caching, database optimization, and autoscaling rather than any single fix.

How to scale from 0 to a million users?

Most systems scale in stages: start with a well-indexed database and a stateless application tier, add caching and a CDN once static content matters, then introduce read replicas, queues, and autoscaling as traffic grows unevenly. Sharding and predictive autoscaling are typically only needed once simpler fixes like indexing, replication, and caching are exhausted.

How do I make an app scalable?

Design for statelessness from the start so any instance can handle any request, then add horizontal scaling, caching, and database read replicas as load increases. Continuous load testing with open-loop methods and SLO-driven monitoring tell you which of those levers you actually need next.

What are examples of scalability?

A CDN caching static assets at the edge so origin servers handle far less traffic is one example, as is a Kubernetes cluster using the Horizontal Pod Autoscaler to add pod replicas when request volume rises. Database read replicas absorbing reporting traffic so the primary database stays fast for transactions is another common pattern.

Sources

Have a project this touches on?

A free 10-minute call is the fastest way to find out whether we are the right studio for it.

Book a free 10-min call

Keep reading