It says it right there in the documentation. But somehow I had never picked up on it.

If you enable Web-garden mode by setting the webGarden attribute to true in the processModel element of the application's Web.config file, do not use InProc session state mode. If you do, data loss can occur if different requests for the same session are served by different worker processes.

We've been using ScaleOut StateServer for some time now. Aside from annoyances like not being able to see performance statistics for the in-memory cache, ScaleOut StateServer is a good product for managing state data in a large web server farm. (Read about my solution to the missing performance counters problem.) We followed most of the best practices for setting up StateServer. But as we approach the 11 million user mark at SnagAJob.com, we have been seeing more and more issues that point to ScaleOut being the root cause of some serious performance issues on our site during peak load.

We were using ten virtual machines running IIS6 with five IIS worker process instances per server. That's fifty worker processes. Not a huge web server farm but not puny, either. In any case, after speaking with ScaleOut support and learning that many of their customers report networking problems while using vmware ESX to host Windows Server and IIS, we decided to go back to InProc session state management for a while. To do this, we had to enable one of the session persistence modes on our load balancers which is far from ideal. Doing that makes things quite a bit less fault tolerant and slows down the deployment process considerably. But we had to do something and this seemed fairly low risk.

When we switched back to InProc session state management mode, we started noticing strange behavior for logged in users. Tests confirmed that the multi-instance web gardens were the culprit. We downgraded to one IIS worker process instance per server and everything is beautiful.

It's completely counter-intuitive to me (and to everyone else I've spoken) that multiple IIS worker process instances operating as a single web garden don't share InProc session state. Logically, it's one server, right? And Microsoft has had support for memory-mapped files in Win32 for more than a decade. Well, as I said, it's right there in the documentation. But an event log warning on application startup would have been nice in this case.

By the way, even though I like ScaleOut StateServer, our fifty IIS worker process instances have been reduced to three (yes, 3) and we are handling our normal volume nicely. We know that our networking environment isn't ideal. Any company that has to be meet the PCI Security Standards is going to jump through some intricate networking hoops. But there's little reason to deploy fifty server instances to solve a scaling problem when three will do.