Never send requests directly to Kestrel

A few days ago, I was tasked with writing an application that would send several requests to an API. It was a ConsoleApp and wasn’t overly complicated. I wrote the application, tested it on my system, and it worked fine. Then, they said the requests had to be sent in parallel! I made a small change and sent all the requests simultaneously. The workflow was like this: I created a list of tasks and then waited for all of them to complete (await Task.WhenAll). I tested the application with 10,000 requests, and it worked. At first, I thought the maximum would be 1,000 or 2,000 requests. Anyway, on the day the application was supposed to run on the server, we went into a meeting, and they said the number of requests would be close to 150,000. I was a bit stressed because I hadn’t tested it at that scale. In that meeting, the infrastructure team lead, the database team lead, the backend team lead, and the CTO were all present. When the application ran, after a few seconds, the application receiving the requests threw an error. (Perfect timing!). It couldn’t have been worse! I felt terrible—even if I had been fired, I wouldn’t have felt this bad. No one said anything to upset me, but I was deeply frustrated that the application had to fail right then! Anyway, we ended up sending the requests in batches of 100 at a time, and the job was done.

I didn’t forget about it and thought I had to figure out why that application failed under such a high request load at that time. I brought the same number of requests to my system, ran it, and saw that the application failed. I searched a lot, and people said you shouldn’t send that many requests at once and things like that. But I wasn’t convinced. Then, I ran the application receiving the requests on IIS.

It was as if nothing was happening—150,000 requests were being sent to it, and it handled all of them! I was really surprised. Kestrel, which everyone praises for being good, having great performance, etc., how could it fail so quickly compared to IIS? It was a bad time—the Iran-Israel conflict was happening, and the internet was practically down. I was searching in a way where I could only rely on the titles and descriptions of sites on Google’s first page to find something. No luck. I asked a question on Stack Overflow, explaining that I had such a scenario and ran into this problem: Why does this code work fine on IIS but not on Kestrel? Again, I got no results. I asked Mr. Vahid Nasiri on his website about this issue, mentioning that I had posted on Stack Overflow and would appreciate any insights he might have.

I don’t know how to thank him enough. He left a comment on my question, saying that when you use IIS, requests are managed by HTTP.sys, and HTTP.sys operates at the kernel level of the operating system!

A lot of new questions popped into my mind—what exactly is HTTP.sys, and how does it work?

HTTP.sys is a driver that operates at the kernel level of the OS and handles a lot of tasks. This very driver gives the application the capability to connect directly to the internet and receive user requests. Meanwhile, they say Kestrel shouldn’t be directly exposed to the internet and shouldn’t receive user requests directly—Kestrel must always be behind a reverse proxy like nginx or IIS. HTTP.sys has many capabilities; you should definitely read about it.

The mistake we made was that the application sending the requests was a Kestrel running on the same server where the receiving Kestrel was hosted. In other words, Kestrel was directly receiving requests from the ConsoleApp. This caused Kestrel to throw a flood of errors after just a few seconds.

The lessons I learned from this experience were:

1. Whenever you develop an application, ask upfront how many requests it’s expected to handle.

2. Determine the scale it’s supposed to operate at.

3. Always, always simulate the scenario at least once before deploying to the production server.

4. When you encounter an error, don’t brush it off—understand why it happened. This alone will help you learn a lot of new things and add to your experience.

Here are some references about HTTP.sys for you to read and expand your knowledge:

• ASP.NET Core Servers

• HTTP.sys in ASP.NET Core

• HTTP.sys vs Kestrel: Pros and Cons

Powered by Froala Editor