05/SCALE AND PERFORMANCE
It gets slow, and we keep buying bigger servers.
The application is fine most of the time and then it is not — under a traffic spike, at month end, or during the one report that everybody runs on Monday.
The response so far has been more resource: a larger instance, a bigger database, another worker. It helps for a while, costs more every month, and the problem returns.
Nobody can point at the actual constraint, because the system is not instrumented well enough to show one.
You need to know what is genuinely limiting throughput before you spend anything else on it.
01/Why it happens
It is nearly always one of these three.
The bottleneck is usually the database, not the compute
Missing indexes, N+1 query patterns, lock contention and connection pool exhaustion. Adding application servers makes several of these actively worse.
Scaling was never actually tested
Auto Scaling exists in the console but has not been exercised. Under real load it turns out to react too slowly, or to add instances that immediately queue behind the same constraint.
Synchronous work sits in the request path
Report generation, third-party calls, email sending and image processing done inline. Each one holds a request open and consumes capacity that should be serving users.
02/What we do about it
The engagement, phase by phase.
Measure under real conditions
about a week
Instrumentation goes in first: request traces, database query timings, resource saturation. Optimising before this step is guessing with extra steps.
Produces
Profiles and traces from production, not from a laptop
Identify the actual constraint
a few days
One constraint dominates at a time. We name it, show the data, and predict what will happen once it is removed — including what becomes the next limit.
Produces
A written finding naming the bottleneck and the evidence
Fix it at the right layer
one to three weeks per constraint
Indexes and query changes, moving synchronous work onto a queue, caching where invalidation is tractable, or splitting a hot path. The layer is chosen by the evidence.
Produces
Targeted changes to queries, code, caching or architecture
Verify against load
a few days
Load reproduced at a level above your current peak, so the improvement is demonstrated rather than asserted, and the next ceiling is a known number.
Produces
A load test showing behaviour before and after
Set up the scaling you actually need
a few days
Scaling policies exercised under load and tuned to react before users feel it — once the underlying constraint no longer makes extra capacity pointless.
Produces
Tested auto scaling, tuned to how the workload behaves
03/What done looks like
How you will know it worked.
- The dominant bottleneck identified, with the evidence written down
- Response times measured before and after, under comparable load
- A known ceiling: the traffic level at which the next constraint appears
- Slow synchronous work moved out of the request path
- Auto scaling that has been tested rather than merely configured
- Dashboards showing the metrics that actually predict a slowdown
04/Honest caveats
Where this gets harder.
Every engagement has failure modes. Here are this one’s, before you commit to anything rather than after.
- Performance work removes one constraint and reveals the next. That is the process working, but it means this is rarely a single fix.
- Some slowness is a data model problem, and fixing it properly means changing schemas and migrating data. We will say when that is the real answer.
- Caching is the fastest win and the most common source of subtle bugs. We use it where invalidation is genuinely tractable and avoid it where it is not.
- If the constraint is a third-party service, our options narrow to queuing, caching and renegotiating with them. We will be clear when that is where we have landed.
05/Questions
Scale and performance, asked about.
01Can you just tell us what instance size we need?
We can, but it is usually the wrong question. Compute is rarely the binding constraint, and sizing up around a database or architectural limit costs money every month without fixing the problem underneath.
02Do you need production access to do this?
We need production telemetry, which is not the same thing. Read-only access and the ability to deploy instrumentation is normally enough to find the constraint.
03How much faster will it get?
Unknown until we measure, and any number offered beforehand is invented. After the measurement phase we can predict the improvement for a specific change with reasonable confidence.
Sound familiar?
Describe your version of it. You will get a reply from an engineer within one business day, including if we think you do not need us.