This post describes a story about my experience in observing latency and performance of two gateway load balancers. It was part of a cloud migration project, we needed to pick up a load balancer for our api gateway in a new environment.

Optimistic ideal aims were set in place, like any engineering project wants to conceive in every corner of the world, we needed a fast and reliable one.

We started out from the first aim which was to catch the fast one by measuring latency, getting gateways performance profiles, and bringing the numbers side by side.

There were two gateways we examined, argo-a and tunnel-b. It would’ve been fun too if I could present them as ng-a and inx-b. To avoid this writing from a commercial use, let’s go with gateway-a and gateway-b.

We provisioned both gateways and exposed our API gateway in Kubernetes for comparison.

Load testing is straightforward with tools such as Vegeta. It summarizes latency and exports raw results for further analysis.

Within a day I had enough data to draft a report for the team.

As blind as it could be, coming from the campaign of getting the fastest one, when l looked at this table, my pupils irresistibly went straight to the min values.

Shifting my sighting to the p50, and p90 values, the difference looked so trivial. Hundreds of milliseconds won’t put a dent in the scale we had. I tripped over the p99 and the max values, the p99 values were still under milliseconds, they didn’t seem alarming.

Gateway B’s max latency was 14 seconds—uncomfortably high, but easy to dismiss as a rare outlier that production might absorb.

I concluded prematurely that gateway B was the better choice and closed the analysis, relieved that the work had moved quickly.

By the next day, the 14 seconds thought sneaked back into my mind. It was this voice. “Is it really? Is this really what’s happening? How can this look so easy? How can this feel so wrong?” These questions ran over my head again and again like a creeping Kubernetes CrashLoopBackOff. Something still felt off.

The outlier kept bothering me until I returned to the raw Vegeta exports.

I looked for a practical way to chart the distribution in Jupyter and ended up in the Seaborn documentation. catplot gave me a compact view of latency spread in a few lines of code.

The chart showed more high-latency requests than the summary table had implied—the 14 s max was not an isolated spike.

That forced me to revisit what percentiles actually represent. I started with the percentile definition on Wikipedia.

In statistics, a k-th percentile, also known as percentile score or centile, is a score below which a given percentage k of scores in its frequency distribution falls (“exclusive” definition) or a score at or below which a given percentage falls (“inclusive” definition).

These words looked like an abstract class with a too broad implementation for me. Suffice to know that I won’t be able to instantiate it, I need to extend and make it meaningful in order to rake its fungible properties in.

How can I map this wikipedia percentile definition to my current circumstance? What does it mean to my data?

Believing like I knew how to do it, I went to assess my thoughts by doing a basic behavioral observation with a brute variable testing. The idea was what if I sampled my data with rudimentary numbers in the shape of my real data in order to excavate the ground.

I went lining up an array of numbers and trying to discern what the p25 and p90 produce. Here’s to demonstrate the percentile calculations.

np.percentile([1,1,2,10], 25)

# 1.0

np.percentile([1,1,2,10], 90)

# 7.600000000000001

Given that they result in these numbers, I set out some low numbers to theme them up with my load test data pattern, and it painted its real colors.

np.percentile([1,1,1,1,1,1,1,1,1,1,2,10], 25)

# 1.0

np.percentile([1,1,1,1,1,1,1,1,1,1,2,10], 90)

# 1.9000000000000004

This conveyed my mind that If I saw dominating values in low percentiles, and there was a population disparity between low and high percentiles, I would see the values of high percentiles approaching low percentiles.

Translating that to my case, If my load test data was populated with plenty of low latency results which gave me a low number of p25 per say, it shouldn’t be shocking if the p99 would come near to that low number.

Consequently, the given sheer amount of low latencies blitzed the p90 and p99 gateway-b latency in a tumbling sense.

It got me thinking, if this was true, would this behave consistently? Was this just a prick of defiance? Would I find it if I do it repeatedly?

I went performing 5 tests iterations for both gateways harvesting more data in light of purging my assumptions.

It was alleviating though unfortunate to see this concerning pattern reemerged in other gateway-b test iterations. This was invisible until I pulled the sns cat plot.

It was kind of working backward, where a visualization quickly helped me out see what was happening in my data, then it lured me into the bottom with statistical percentile observation.

I had initially favored gateway A, so I checked the data carefully to avoid confirming that bias.

I went on my hunt carefully. I deliberately allocated breaks just because I didn’t want to get caught in a train of one sided dubious complacent thoughts. What a very dangerous path to be on.