Turning a misleading p95 into trustworthy production observability

We ran 200 virtual users against Verachi in production using one isolated account. The run surfaced connection timeouts, but the application did not show corresponding CPU, memory, or event-loop saturation. Before changing capacity, we had to establish which measurements we could trust.

That review found two observability defects. Request-duration histograms used buckets that did not match their seconds-based unit. The error-log panel also selected a service name that the live web process did not emit. A plausible latency number and a quiet log panel were therefore incomplete evidence.

We corrected the telemetry, added the missing edge and runtime signals, verified every query against live data, and then made the smallest capacity change supported by the result: two web replicas and two tunnel replicas for Verachi. We also moved Midflight to two web replicas; its tunnel already ran with two.

The test and its boundary

The k6 workload used 200 virtual users sharing one authenticated Verachi account. Login happened during setup, outside the steady-state loop, so the test measured authenticated product reads instead of the login rate limiter. Health checks also ran outside the loop.

A same-account test is useful for finding shared-session, routing, cache, and read-path limits. It is not a model of 200 independent customers. It does not prove multi-account isolation, sustained write capacity, or streaming capacity. We treat those as separate profiles because they stress different parts of the system.

A p95 is only as accurate as its buckets

Verachi records HTTP and database durations in seconds. The metric pipeline had not defined explicit buckets for seconds-based histograms. Prometheus could still calculate a quantile, but the bucket boundaries were too coarse for the latency range we needed to inspect.

We added one OpenTelemetry histogram view for every instrument whose unit is s. Its boundaries run from 5 milliseconds to 10 seconds. This keeps the unit contract in the telemetry layer, where every caller benefits, instead of compensating in one Grafana query.

0.005, 0.01, 0.025, 0.05, 0.075, 0.1,
0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10

A focused post-deployment probe sent 40 authenticated requests to /api/auth/me. The corrected histogram reported an 18.3 millisecond p95. That number verifies the metric path and its unit. It is not a capacity benchmark.

The missing failures were at the edge and in the label contract

Application HTTP metrics begin after a request reaches an instrumented handler. A connection timeout before that point cannot appear as an application 5xx. The original dashboard made that boundary hard to see.

We added Cloudflare tunnel request rate, error rate, response status, and active high-availability connections. These panels sit beside application request rate, status classes, and latency. An operator can now tell whether pressure stopped at the edge or reached the application.

Loki exposed a second contract error. The dashboard queried the web service as verachi-web, while production logs identify it as verachi. Worker and tunnel labels were already correct. The updated selector uses the labels present in Loki, and a source check now prevents the stale selector from returning.

We scaled replicas, not resource limits

The 200-VU run showed connection pressure. The application resource signals did not show CPU, memory, or Node.js event-loop saturation that justified larger pods. Increasing every limit would have spent capacity without addressing the observed boundary.

We increased Verachi from one to two web replicas and from one to two tunnel replicas. We increased Midflight from one to two web replicas; its tunnel already had two. Existing per-pod limits stayed in place: one CPU and 1 GiB for Verachi web, and one CPU and 768 MiB for Midflight web.

Table 1. Production replica and resource-limit changes. Worker replica counts were outside this change.

Verification after deployment

Verachi released as v1.1.158 and Midflight as v1.0.161. Argo CD reported both applications synchronized and healthy. Kubernetes reported two desired, ready, and available web replicas for each product. Verachi also reported two ready tunnel replicas.

We then generated a small authenticated request sample and queried the production data sources directly. Request rate, p95 latency, status classes, database operations, shared Redis hit ratio, CPU, memory, event-loop utilization and delay, replica availability, restarts, and logs all returned data. The verification window showed eight active Verachi tunnel connections and no current tunnel error rate.

These observations prove that the dashboard is wired to live data. They do not prove that the new replica count sustains 200 VUs. That requires another controlled run against the released configuration.

What we learned

A dashboard is part of the system it describes. Its unit contracts, label selectors, and network boundaries require the same review as application code. A panel returning a number is not sufficient. We need to know where the number originates, which failures it excludes, and whether a real request changes it.

The same rule applies to capacity. We did not make the pods larger because the available evidence did not support that change. We added horizontal redundancy at the two boundaries implicated by the test, retained the current resource limits, and left a staged 25, 50, 100, 150, and 200-VU profile for the next measurement.

What's next

The next production run will repeat the same-account capacity profile against the two-replica release. Separate multi-account, write, and streaming profiles will remain separate evidence. We will increase CPU or memory only when saturation, throttling, or latency correlated with those resources appears in the same test window.