Simon Mourier's HtmlAgilityPack.HtmlWeb exposes three events:
- PreRequest
- PostResponse, and
- PreHandleDocument
I make use of all of them in the following reworking of last night's
post. I thought about doing something with HttpWebRequest or WebClient,
but then I realized that if I'm going to use HtmlAgilityPack, HtmlWeb,
and HtmlDocument, I might as well exploit the exposed event model to
the fullest. The result, DigitalCertificateHtmlWeb, is a more general
purpose class that will intercept the result from a redirect, allow
inspection, and then allow modification of the returned HtmlDocument
object without out any work by the consuming client other than
attaching an event handler to the PreHandleDocument event.
The code makes use of .NET 2.0's anonymous methods and delegates to accomplish some elegant recursion.
- When dweb.Load(...) is called, the first event handler is
PreRequest. Here, the digital certifcate is attached to the outbound
HttpWebRequest.
- Next, the PostResponse event is raised, and the ResponseUri
is fetched from the HttpWebResponse. This is important for the delegate
later to know whether the original target handled the request or
whether a redirection occurred.
- Next, the PreHandleDocument event, the only one that the
client code need specify, is raised. This code checks the previously
acquired LastResponseUri property to see whether the path matches the
custom challenge-response redirection URL.
- In the first run, it does match, so the if block is
executed. This uses the methods of HtmlAgilityPack.HtmlDocument, the
passed in argument, to fetch some FORM values and apply them to a
NameValueCollection.
- When the NameValueCollection is filled, the SubmitFormValues method is called. This the most interesting part.
- Within SubmitFormValues, an anonymous delegate method of type
PreRequestHandler is created. This anonymous method is a closure, which
freezes the state of the passed in fv argument. This method does not
execute immediately, of course. It is actually added to the event
handlers list for the PreRequest event, becase as we've already seen,
we already have a PreRequest handler that attaches the digital
certificate to the outbound request.
- Continuing, the code “this.PreRequest += handler;
HtmlDocument doc = this.Load(url, “POST“); this.PreRequest -=
handler;“ instructs the class to append that event handler, load the
document, then remove the event handler as soon as the Load operation
completes.
- Stepping in the debugger from this point now shows that the
first PreRequest handler executes, attaching the certificate again to
the outbound request. The next statement to run is the newly appended
PreRequest event handler, the anonymous delegate we just created.
- After this, the trusty old PostResponse handler, the one and
only, is invoked after the PostResponse event is raised. It reassigns
the LastResponseUri, but this time the URI is equal to the originally
requested URI, because the custom authentication FORM has been
submitted, and the cookies from the server have been digested.
- Finally, the PreHandleDocument event is raised again, and
this time the check for the AbsoluteUri.Contains(...) fails, so the
else clause executes.
using System;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Text;
using System.Net;
using System.IO;
using HtmlAgilityPack;
using Microsoft.Web.Services2.Security.X509;
public class DigitalCertificateHtmlWeb : HtmlWeb
{
private X509Certificate _cert = null;
public X509Certificate Certificate
{
get { return _cert; }
set { _cert = value; }
}
private Uri _lastResponseUri;
public Uri LastResponseUri
{
get { return _lastResponseUri; }
set { _lastResponseUri = value; }
}
public DigitalCertificateHtmlWeb()
{
this.UseCookies = true;
// Create a PreRequest handler to attach the ceriticate
// to the outgoing request
this.PreRequest += delegate(HttpWebRequest request)
{
// Attach the digital certificate to the outbound request
request.ClientCertificates.Add(_cert);
return true;
};
// Create a PostResponse handler to expose the most
// recent ResponseUri object to client objects
this.PostResponse += delegate(HttpWebRequest request,
HttpWebResponse response)
{
this._lastResponseUri = response.ResponseUri;
};
}
public HtmlDocument SubmitFormValues(NameValueCollection fv, string url)
{
// Attach a temporary delegate to handle attaching
// the post back data
PreRequestHandler handler = delegate(HttpWebRequest request)
{
string payload = AssemblePostPayload(fv);
byte[] buff = Encoding.ASCII.GetBytes(payload.ToCharArray());
request.ContentLength = buff.Length;
request.ContentType = "application/x-www-form-urlencoded";
System.IO.Stream reqStream = request.GetRequestStream();
reqStream.Write(buff, 0, buff.Length);
return true;
};
this.PreRequest += handler;
HtmlDocument doc = this.Load(url, "POST");
this.PreRequest -= handler;
return doc;
}
private string AssemblePostPayload(NameValueCollection fv)
{
StringBuilder sb = new StringBuilder();
foreach (String key in fv.AllKeys)
{
sb.Append("&" + key + "=" + fv.Get(key));
}
return sb.ToString().Substring(1);
}
public static X509Certificate GetCertificate(string filter)
{
X509CertificateStore store =
X509CertificateStore.CurrentUserStore
(
X509CertificateStore.MyStore
);
store.OpenRead();
X509CertificateCollection certs =
store.FindCertificateBySubjectString(filter);
if (certs != null && certs.Count > 0)
{
X509Certificate cert = certs[0];
return cert;
}
else
{
return null;
}
}
}
public class TestDriveDigitalCertificateHtmlWeb
{
public string Execute()
{
DigitalCertificateHtmlWeb dweb = new DigitalCertificateHtmlWeb();
dweb.Certificate = DigitalCertificateHtmlWeb.GetCertificate(“uvconsulting@wdevs.com“);
dweb.PreHandleDocument += delegate(HtmlDocument doc)
{
if (dweb.LastResponseUri.AbsoluteUri.Contains("www.topsecretsite.com"))
{
// Create a collection to hold the form values that will
// need to be posted back to the server for authentication
NameValueCollection fv = new NameValueCollection(7);
// Add the User token value to the collection
HtmlNode d = doc.DocumentNode.SelectSingleNode("//input[@name='USER']");
fv.Add("USER", d.Attributes["value"].Value);
// Add the smagentname value
d = doc.DocumentNode.SelectSingleNode("//input[@name='smagentname']");
fv.Add("smagentname", d.Attributes["value"].Value);
// Add the SMENC, SMLOCALE, and smauthreason values
fv.Add("SMENC", "ISO-88591");
fv.Add("SMLOCALE", "US-EN");
fv.Add("smauthreason", "0");
// Add the challenge phrase
fv.Add("password", "***top secret***");
// Specify the target URL
d = doc.DocumentNode.SelectSingleNode("//input[@name='target']");
fv.Add("target", d.Attributes["value"].Value);
// Make another request ...
HtmlDocument newDoc = dweb.SubmitFormValues(fv, “https://www.topsecretsite.com/login.aspx“);
doc.LoadHtml(newDoc.DocumentNode.OuterHtml);
}
};
HtmlDocument hdoc = dweb.Load(“https://subdomain.topsecretsite.com/ping.aspx“);
return hdoc.DocumentNode.OuterHtml;
}
}
I tried this variation also:
HtmlDocument hdoc = dweb.Load(“https://subdomain.topsecretsite.com/ping.aspx“);
string rv = hdoc.DocumentNode.OuterHtml + "\n\n";
hdoc = dweb.Load(“https://subdomain.topsecretsite.com/ping.aspx“);
rv += hdoc.DocumentNode.OuterHtml;
return rv;
I expected in this case for the second request to go directly
through without being intercepted by the custom authentication
redirect. I thought that the UseCookies assignment would persist the
cookies across calls, but I'm not sure it's working that way, because
the whole sequence takes place the same way again.
------
OK, as soon as I posted a question to Simon's blog, I figured out
what was wrong with the cookies. I needed assign a CookieContainer in
the callback. That enabled persistence of the same collection across
calls to HtmlWeb.Load
public class DigitalCertificateHtmlWeb : HtmlWeb
{
private X509Certificate _cert = null;
public X509Certificate Certificate
{
get { return _cert; }
set { _cert = value; }
}
private Uri _lastResponseUri; // to check for redirection
private CookieContainer _cookies = new CookieContainer();
public Uri LastResponseUri
{
get { return _lastResponseUri; }
set { _lastResponseUri = value; }
}
public DigitalCertificateHtmlWeb()
{
this.UseCookies = true;
// Create a PreRequest handler to attach the ceriticate
// to the outgoing request
this.PreRequest += delegate(HttpWebRequest request)
{
request.CookieContainer = _cookies;
// Attach the digital certificate to the outbound request
request.ClientCertificates.Add(_cert);
return true;
};
// Create a PostResponse handler to expose the most
// recent ResponseUri object to client objects
this.PostResponse += delegate(HttpWebRequest request,
HttpWebResponse response)
{
this._lastResponseUri = response.ResponseUri;
};
}
...