Http.sys in WinXP SP2: What It Means with Windows Communication Foundation

Windows Communication Foundation (WCF) and the HTTP API

When Windows 2003 Server was released, Microsoft introduced the HTTP API and the Kernel mode Http.sys driver. With the W2K3 release, the first applications integrated with the new API was IIS 6.0 and consequently ASP.NET based applications. The HTTP API significantly raised the performance of HTTP server based applications. Some of the performance benchmarks and general background on IIS 6.0 and HTTP API capabilities and programming considerations are listed here:

http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/webapp/iis/iis6perf.mspx .

So, that was great for the Server side applications. Microsoft then included the HTTP API as part of the Service Pack 2 (SP2) update for Windows XP. For the most part, it sat dormant, unused on the XP platform, the client side of distributed applications. Microsoft has never updated IIS, it’s still version 5.1 and doesn’t leverage the HTTP API. For the most part, it’s not entirely clear why you would need such a server oriented capability on a desktop version of an operating system. That is until Windows Communication Foundation ("WCF" - f.k.a. Indigo) was introduced. Later in this article we’ll see why it makes a difference with WCF and Duplex Message Exchange Patterns.

Background

At the time of Windows 2003 Server release, ASP.NET had become the primary model for development and deployment of HTTP based applications - both Web Browser and Web Service based applications. Given the performance capabilities of the new HTTP API, ASP.NET with the Windows worker process model architecture leveraged it as well. This architectural advantage provided ASP.NET significant performance advantages by running on Windows 2003 Server as opposed to running on Windows 2000 and Windows XP.

The heart of this architecture change for the HTTP API is the Http.sys Kernel mode device driver. Historically, applications written for the HTTP protocol have run in what is known as User mode. These are processes that have to deal with preemptive things such as software interrupts, context switches, thread scheduling, and generally, restricted access to system resources. In Kernel mode, the encumbrances are minimized. But writing Kernel mode applications is risky and difficult.

In the past, HTTP servers, such as IIS, Apache, and other applications on Winodws, have relied on the Winsock API running in User mode for creating what is known as a network listener. Network listeners within an instance of an Operating System generally get exclusive ownership of an IP port on a per-application (or thread) basis. What this means is that only a single application can "listen" on IP Port 80 on a machine at a time. Historically, this hasn’t been that much of a problem as most machines usually have a single application providing services for each protocol. If you look at the file services located on windows systems in %windir%\system32\drivers\etc\ you’ll see a list of "Well-known" services and their respective IP Ports and Protocol types (either TCP or UDP). Applications that require these ports get sole ownership of the IP port if that application or service is running. To demonstrate how this services file is used for lookup and determining the IP port of a service; enter the following from a Command Prompt

        telnet www.msn.com http <enter>

At this point you should get a blank screen - then type (you must be connected directly to the internet, not through a proxy server, for this to work):

        GET / <enter>

You should receive something like the following:

S: A12
X-AspNet-Version: 1.1.4322
Location: http://st.msn.com/br/gbl/htm/err/500er.htm?aspxerrorpath=/
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 175
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href='http://st.msn.com/br/gbl/htm/err/500er.htm?aspxerrorpath=/'>here</a>.</h2>
</body></html>
Connection to host lost.

HTTP API & Http.sys

So, what is the Http.sys Kernel mode driver providing for you? First let’s take a quick look at the inter-process architecture of how it looked before and after Http.sys.

Figure 1: IIS 5 Process Model

Figure 1 illustrates how IIS 5 runs Http based applications in User mode and leveraging the Winsock API. The immediate impact is the additional overhead associated with User mode processes and encumbered access to system resources. The other, as will be explained later, is only one application can listen on an IP port at a time.

Figure 2: IIS 6 Process Model

In Figure 2, the Worker Process Isolation model is in place, leveraging the HTTP API and Http.sys directly - allowing the management of the HTTP handshake and request management directly at the Kernel level with fewer encumberences.

To be clear, the Kernel mode driver doesn’t allow applications to execute in Kernel mode; what is does provide is routing of requests and response through the Kernel into an application’s HTTP API request queue the onto the responsible application code in another process that provides the service in support of the requests for a specific Uri. Both ASP.NET 1.1 and 2.0 make use of this in the Worker Process model on Windows 2003 and gain the advantage of the HTTP API. In the future, IIS 7 will add what is known as Windows Activation Services ("WAS") but that’s better left to it’s own article.

Again, applications that process the requests are all still running in User mode, it’s just the initial request handling (socket management, caching, etc.) is handled at the Kernel level before handing off to the application.

What does this buy you?

Caching - static content is cached Kernel level providing even greater response
Logging Support - this is the IIS log capability that is now streamlined and faster
Bandwidth Control - allowing greater scalability control and throttling
Reliability - queued requests pend in Http.sys, not the process that will handle it - so if a process is recycling it doesn’t get lost
IP Port re-use - more than one application can listen on Port 80 - or any IP port leveraged within the Http.sys Kernel mode driver!

While the listener and request queuing support is in a shared Kernel mode driver applications running in User mode still have process isolation for the application code. This provides the protection if one Application Pool is impacted by a site, other Application Pools don’t go to the trash as well. Again, good background in IIS 6 is here: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/webapp/iis/iis6perf.mspx

How to Leverage in your applications

The easiest way to leverage the HTTP API is to write ASP.NET applications that are hosted in IIS 6 on Windows 2003 and just deploy. Remember, you get nice Server side applications that can scale and support a greater level of performance than you can with the IIS 5 Process model.

What happens if you want to create your own hosting application outside of IIS 6, or even provide an application that runs on a client machine, but provides callback capabilities from a server to the user’s workstation? Well, to do that in Windows, you need to leverage the HTTP API. In .NET 1.1, you had to write your own wrappers using PInvoke for the HTTP API (see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/http/http/http_api_start_page.asp for details).

With .NET 2.0, the Framework introduced the HttpListener Class, a direct wrapper around the HTTP API providing a nice .NET abstraction for creation of your own servers that listen for Http traffic. This capability is only supported on Windows 2003 and Windows XP SP2. So, if you wanted to leverage this capability you needed to deploy to Windows 2003 - even in development environments.

The availability of the HTTP API was strictly Windows 2003 - that is, until XP SP2 when Microsoft introduced the HTTP API to this version of Windows. While it was added to the core OS, IIS remained the same, version 5.1 and still didn't use the HTTP API. So, in order to leverage the HTTP API on Windows XP SP2, you still needed to drop down to the Win32 API to use it or leverage .NET 2.0 HttpListener. That is, until Windows Communication Foundation (WCF - f.k.a. Indigo) arrives.

In comes WCF

WCF makes direct use of the HttpListener class for Http channel bindings, and consequently, leverages the HTTP API capabilities directly. Well, why does anyone care?

Let’s take a scenario of an application that is deployed leveraging what is know as a Duplex Message Exchange Pattern. In that pattern, both server and client respond to network requests. This is comparable to using .NET Remoting with a callback interface. There’s a Duplex sample as part of the upcoming Windows SDK (WinFX) code samples that demonstrates this Duplex pattern.

In WCF and Duplex Contracts, when the client initiates an operation on the Service interface, the framework also provides a "pointer" to a callback interface that is basically hosted on the client side. This callback Uri is defined in a configuration file as a client interface binding address. The framework provides this to the server so that when the server needs to callback into the client it leverages this address and binding for the http call. This Duplex Message Exchange pattern requires 2 HttpListeners on the network - one on the server for the initial operation and one on the client machine for the callback operation.

Figure 3: Duplex over Http Message Exchange

With a Duplex Message Exchange, the client initiates a call, usually leveraging a Proxy class generated from the Service Contract Metadata. As part of the initial call a client interface's binding information is provided on that request to the WCF server. How the Binding is exchanged is buried inside the WCF Framework implementation, but for the most part, the developers that need to leverage the capability just need to implement an interface generated from the Service Contract Metadata (using WCF's SvcUtil.exe utility) and provide the correct information inside of the applications configuration file's service.model section.

Figure 4: WCF Application Configuration for Duplex Client Support

Why does it matter?

With WCF and down-level support for Windows XP SP2, clients can now leverage the capabilities of HTTP API. Why is this important? Well WCF introduces a simple implementation model for "Callback" capabilities with Duplex service contracts and communications. Once such implementation exists in System.ServiceModel.WSDualHttpBinding (shown in the Application Configuration file above) binding that is part of WCF (note the namespace or type may change with the shipping version of WCF/WinFX).

This allows distributed applications that are running a WCF client on a client workstation to communicate with a Service tier, and provide a callback channel. Since it’s expected that multiple applications will exist on each client workstation, this allows each application to share an IP port, but different Uri’s as they are registered with the Http.sys Kernel level driver.

Why is that important? Well it allows a minimal workstation modification to firewall and IP transport policies for all applications leveraging WCF. Generally, out of the box, Windows XP SP2 with the Windows Firewall running (and most firewalls for that matter) limits all unsolicited inbound communications on IP ports. Each IP port must be "opened" and IP scope polices applied.

So, prior to this capability, each Application publisher must establish a unique IP port for their application, ensuring they don’t step on well known ports or application ports that may be in use on workstations by other applications. So, how do you do this in an organization that has 50,000+ people and a mix of applications that may or may not be using the port you want to use? You might say - "No problem, just make it a config file setting!"

Only problem is, what happens if the 1st 100 users are not using any applications that consume port 8080. You do a roll-out, Then, you have to roll-out to an additional 1,000 users and those user's workstations already have port 8080 in use? Lots of configuration management issues.

Now, we can re-use these limited IP ports across all HTTP API based applications. This doesn’t eliminate the initial potential conflict with already deployed applications that don’t leverage HTTP API, it just alleviates any future conflicts.

Quick Demonstration

The following is a quick demonstration application illustrating how WCF interacts with HttpListener and ultimately, the HTTP API and Kernel model Http.sys for registration listening endpoints.

Example Code Solution here

This link is a VS 2005 Sample that leverages the December 2005 CTP version of WinFX (http://msdn.microsoft.com/windowsvista/). The lists of requirements for the WinFX December 2005 are listed there. In order for this sample to run you need:

.NET 2.0 Runtime or SDK
WinFX December 2005 Runtime Components
Windows XP SP2 or Windows 2003 SP1

Note, that in order to see the following set of messages, you must run the example outside of Visual Studio. Otherwise, the VS debugger will "step" in and just makes getting to the "point" of this article that must harder.

Once the Zip file is unpacked, navigate to the following folder:

   ..\HttpSysDemo\ServiceHost\bin\Debug

In that folder, double click on MyServiceHost.exe - the following should appear:

Figure 5: WCF Service Host first instance started

Once you have the service host running, click on the first button "once". You should see in the status bar that "Host 1 - started".

Now, click the same button again - this will attempt to create another WCF Service host at the same Url.

Figure 6: Duplicate Service Host Uri registration in same Process

What’s happened here is WCF’s channel factory has thrown an exception indicating that within this Application process (AppDomain), you’ve already registered the same Url - something that isn’t supported and throws a System.InvalidOperationException type exception.

So, at this point, nothing special. Just click the "OK" button to dismiss the Message Box.

Next, startup up another instance of MyServiceHost.exe so on your desktop you have 2 instances of MyServiceHost.exe running. It should look like the following:

Figure 7: Service Host Second Instance Not Started

Note that the first instance should still show that " Host 1 - started" and the second instance shows "Host 1 - stopped".

Now, in the second instance, where the status bar indicates "Host 1 - stopped", click on the first button to startup another Listener that will, through the WCF framework, attempt to register with HTTP API.

Figure 8: Duplicate Service Host Instance - Difference Process

Here we get a different exception - something clearly indicating that the HTTP API is being used. This scenario throws a System.ServiceModel.AddressAlreadyInUseException exception. Clearly indicating in the exception message that another application has already reserved and "locked" the Uri with HTTP API.

So, what’s so different about this from just any old application registering itself on an IP port? Well, in the second instance of MyServiceHost.exe, click the second button labeled "Start Host Service 2". You should now see the following:

Figure 9: Multiple Service Hosts on Difference Processes Sharing IP Port

Here we have 2 distinct Windows Process listening on the same IP port - something that couldn’t be done prior to HTTP API.

Note: Port 8080 is used in this example - IIS 5.1 on XP generally uses port 80 - default for http traffic

There's one additional note on the Uri that is passed down to the HTTP API. The HttpListener class in .NET 2.0 supports the Uri naming convention of the HTTP API. What that means is you can register the following directly with the HttpListener (which is direct to HTTP API) and not get any exceptions:

http://127.0.0.1:8080/service1
http://myMachine:8080/service1

Now, the above 2 Uri's will work with no exception thrown by .NET 2.0; however, in WCF, every address (at least this is how it works in the December 2005 CTP) is converted to the following:

http://+:8080/service1

What that means is "listen on all machine IP addresses at port 8080". So, with WCF, I can't listen on different IP addresses with the same relative Uri; I'm forced to listen on all local addresses with the current implementation. I still get to vary the non-server part of the Uri and listen on the same IP port.

Conclusions

So, the HTTP API provides a Kernel level driver (Http.sys) that allows "Host Header" and Uri routing of requests to the appropriate applications without requiring the creation of different virtual servers listening on different ports.

This capability is leveraged by WCF and becomes a valuable feature that can make overall configuration and expectations of clients of these applications much easier to work with. This can alleviate much of the overall configuration aspects for system administrators when deploying more complex applications that require such things as Duplex Message Exchange Patterns.

Hopefully this article helps clear up some parts of HTTP API, the Kernel level driver (Http.sys), how it’s now usable in Windows XP SP2, and some of the reasons as to why it makes a positive difference in both the development aspect of applications, but in the deployment and management in a organization.