An open letter to Senator Anne Cools about 90sAREover and the Richard Warman allegation

Dear Senator Cools,

There has been some buzz in recent days about the testimony of Professor Robert Martin of the University of Western Ontario regarding Human Rights Commissions, which (if you haven't heard it already) can be heard here (loading can be slow, so be patient).

Professor Martin, who spoke in his testimony of his high regard for you and noted that he regarded you as a dear friend, seems to have been especially exercised by allegations that Richard Warman was responsible for a nasty racist screed about you -- so nasty, that he wouldn't (quite rightly) repeat its words. (Nor will I.)

The problem with Martin's testimony is that Warman didn't write that post. He has denied the matter under oath and has sued for defamation those who have made the allegation in unprivileged circumstances. Moreover, the allegation was debunked over a year ago, largely on the internet (from where all these allegations began), including at my site.

One of the basic facts, as you may know, is that that the racist post in question was made by someone using the pseudonym "90sAREover". The allegation that this was Warman depends on the claim that "90sAREover" had the same IP as "lucy" (Warman's pseudonym) and a computer with the same set-up.

• the IP in question, however, was a web-caching proxy shared by most Rogers customers, as I have shown in a post called "Why there is room for doubt that Richard Warman wrote the Cools post". To summarize a long and technical argument, 90sAREover could have been any Rogers customer, or anyone with access to the computer of any Rogers internet customer. This could have been millions of individuals.

• It has also been claimed that Warman's computer was set-up identically to the one that made the racist post. This would not be especially helpful even if true: something over 10% of users had their computers set-up that way (see here), and 10% of millions is still hundreds of thousands. The set-up, however, was not identical. In a post called "why Richard Warman is innocent" , I demonstrate that the computer used to write the racist post was different in at least one key respect from Warman's computer, with the result that they must have been computers.

The technical argument of my posts may be unintelligible to you, and there is no reason that you should believe me -- an anonymous guy on the internet. If you forward this letter to someone knowledgeable in these matters they will confirm to you that there is no good evidence that Warman made the post in question. (That the IP was a proxy and therefore represents a potentially vast pool of users has been endorsed, for example, by the Conservative blog Catprint in the Mash.)^*

During his testimony, Prof. Martin said that you personally were greatly agitated at Warman for these remarks. You have every right to be agitated about this post. But not at Warman, who did not make it.

Obviously, if you have any questions that I can help with, I'd be happy to do so.

Best regards,

Buckets

cc:ReidS@parl.gc.ca, SilvaM@parl.gc.ca, ThilaE@parl.gc.ca, SweetD@parl.gc.ca, HiebeR@parl.gc.ca, MarstW@parl.gc.ca, CotleI@parl.gc.ca

Why there is doubt the Warman wrote the Cools post, Why Warman is probably innocent of the Cools post, Bernard Klatt

Edited to clarify catprint's analysis.^*

Load-balancing in action: did Richard Warman and 90sAREover really have the same IP?

Below and to the right is an excerpt from the Freedomsite's logs, truncated on three sides to make it easier to read. (You can see an untruncated version here on p. 35.)

The top two entries (both with "POST") are related to the notorious Cools message: the first confirming the spelling of "nigger"; the second submitting the nasty message that is at the center of the present controversy. The bottom-most is the last entry for 90sAREover in the logs (or at least those that have been released).

The point to note is that 90sAREover did not have just one IP during his brief visit to the freedomsite message board, but two: 66.185.84.204 and 66.185.84.200, and in the space of three minutes he switches from one to the other at least four times.

I say "at least" four times here because it is important to remember that we are only seeing a portion of what was going on. These two IPs are both Roger's web-caching proxies -- their host names are (respectively) wc09- and wc04.mtnk.rnc.net.cable.rogers.com, showing the tell-tale wc (which abbreviates web-cache) -- and since they are caches there will have been traffic between them and 90sAREover's computer that is not forwarded to the visited site.

What does all this mean? The switching between these two web-caches is another example of load-balancing, a practice by which Rogers shifted traffic among its 42 web-caches to deliver the most efficient service to its subscribers. An attempt to sort out how all of this worked resulted in the string ball to the right (explained here), which traced shifts between proxy-IPs that can be identified in the years 2002-5. Each string in the ball is an example of a load-balancing shift.

This load-balancing provides another wrinkle in argument of identifying 90sAREover. It seems that Rogers' subscribers had what might be called a "home" proxy, which functioned as their default web-cache, and that they would rerouted to a another proxy only when needed, especially (one assumes) at peak hours. But when he made the racist Cools-post, was 90sAREover's "home" proxy 66.185.84.204 and 66.185.84.200 a temporary IP to which he was shifted through load-balancing? Or was his home proxy 66.185.84.200? Or might his "home" have been one of the other web-caches in the series, with the traffic rerouted into 200 and 204?

One of these things is not like the others: why Warman is innocent

Much of Canada's right-wing blogosphere is aflutter with the idea that human rights lawyer Richard Warman might have posted a racist comment about Canadian senator Anne Cools on freedomsite.com, a white-rights forum. The evidence? Freedomsite's logs preserve data from some visits of "lucy" (a handle Warman is known to have used) and from 90sAREover, who had posted a hasty racist screed against black senator Anne Cools. According to freedomsite's expert witness Bernard Klatt, both the IP and the user-agent data were identical for "lucy" and for "90sAREover". Until now we have had to take Klatt's word on this.

Recently these logs have become available for scrutiny (they were included in Klatt's affidavit, which I've recently posted here). The logs make it clear, however, that there are two important differences between the computer used during Warman's visits and those of 90sAREover (the Cools poster); both of these differences are exculpatory of Warman. Here is the evidence in reverse chronological order (compare what is underlined in red).

The log of Warman's visit from Nov. 23, 2003 (p. 33 of Klatt's affidavit):

The log of Warman's visit from Nov. 15, 2003 (p. 30 of Klatt's affidavit, here):

The log of Warman's visit from Nov. 11, 2003 (p. 29 of Klatt's affidavit, here):

The log of Warman's visit from Oct. 15, 2003 (widely cited, e.g. here):

All of Warman's visits from Oct. 15 — Nov. 23, 2003, show the same IP (66.185.84.204 = wc09.mtnk.rnc.net.cable.rogers.com; see the blue underline) and the same operating system and browser (MSIE 6.0; Windows 98 — see the red underline — which points to a generic version of Explorer 6.0 and Windows 98).

Here is the log of 90sAREover of Sept. 5, 2003 (p. 34 of Klatt's affidavit, here). Here the browser details are different (again underlined in red:

Although both Warman and 90sAREover have the same IP and operating system, 90sAREover's user agent string has two key differences from Warman's computer-setup. First, 90sAREover had a customized version of Explorer 6.0 installed, which had been supplied by Rogers (as is signaled by the phrase "Rogers Hi-Speed Internet"); second, he did not actually use IE 6.0 to post, but instead was using RealOne Player version 2 (hence "R1 1.3": see here). None of Warman's log entries, by contrast, show any sign of RealOne, and for all four of his visits his browser was recorded as a generic version of Explorer, not a Rogers one.

These differences are important and exculpatory and overthrow the technical argument for identifying Warman as "90sAREover", which (given that the IP used, 66.185.84.204, could have been anyone of millions of Rogers customers) has now collapsed. The computer that posted the Cools post was set up differently than the one Warman was using three months later.

(Re-written for clarity and context.)

Revisiting the lucy-90sAREover identification

Those who have been following the controversy about the Cools post will probably know the following chart. It apparently imitates one that Bernard Klatt presented at the CHRC hearing -- Klatt refers to such a chart during his testimony -- but this version (with my mark-up in red for point-by-point reference) was posted freedominion.ca on January 20 by "Entropy Squared". It is now part of Warman's statement of claim.

The chart was designed to make a case that Richard Warman was responsible for the posting of a racist denunciation of Senator Anne Cools on Sept. 5, 2003. The evidence argues against that conclusion. (Again, what is in red is my mark-up.)

A ~ a and B ~ b. Warman registered at Freedomsite.org using the pseudonym "lucy" and the IP 66.185.84.204, according to Klatt's testimony. Klatt also testified that 90sAREover (who wrote the notorious Cools post) had the same IP. I have shown here, however, that this IP is one of Rogers' web caching servers and could be used by any Rogers' subscriber -- indeed, the post could have been made by anyone with access to a Rogers-connected computer. This could be any one of millions (here).

• Update. Actually, the IP of "90sAREover" is less clear in the logs than Klatt's testimony implies: in the few minutes that 90sAREover spent at the site, his IP can be seen shifting back and forth between 66.185.84.204 and 66.185.84.200 five or six times. See here.

C ~ c. According to Klatt's testimony, the user-agent string recorded in the logs freedomsite.org show that both "lucy" and "90sAREover" used Windows 98. According to W3schools.com, 12.1% of computers were using Windows 98 in Sept. 2003, when the Cools screed was posted. 12% of millions (see previous), however, is still hundreds of thousands.

D ≠ d. During his testimony at the CHRC, Klatt asserted that "lucy" and "90sAREover" used the same browers. This is wrong. Details of a visitor's browser are found in the user-agent string, which is recorded by many sites. To judge from Klatt's testimony (here)*, the user-agent string of 90sAREover included the phrases "Rogers Hi-Speed Internet; (R1 1.3))". The R1 1.3 means that 90sAREover was not using Explorer as his browser, but in fact was using RealOne Player (version 2). The reference to "Rogers Hi-Speed" shows that the Cools poster had a version of Explorer that had been customized by Rogers for its customers (here), even though he was not using it to visit this site. To judge from Klatt's testimony, none of Warman's own visits show either the RealOne Player or the Rogers-customization. This argues against identification. (One wonders why there are not copies of these logs floating around the internet -- is it because they exonerate Warman?)

*Update. Now that the site logs are available for inspection, rather than Klatt's testimony, it is clear that Richard Warman's browser set-up was different than 90sAREover's.

E ≠ e. The IDs and emails are different. This can hardly be used as an argument that they're run by the same individual. Indeed, those trying to argue that they are identical need to explain why Warman registered "lucy" in mid-November (an account that he never posted from) if he already had the 90sAREover accout. If he were "90sAREover", why wouldn't he simply have used that account rather than registering the new one?

F ~ f. It is a matter of public record that "lucyaubrack@yahoo.ca" was Warman. We don't know, however, whether the "rob_m_simpson@hotmail.com" account was anonymous or whether it reflects someone's actual name.

G ~ g. Both "lucy" and "90sAREover" found their way to freedomsite.org, which was relatively obscure. This no more requires that these two were identical, however, than it would for any of the site's others users, or indeed for visitors of other obscure sites (such as the one you are visiting now). In any case, it is worth noting that 90sAREover is member #1331 (affidavit, p. 10), and "lucy" (Warman's handle) is member #1379 (affidavit, p. 10), which implies that at least 50 new members joined up in this period. These surely aren't all the same individual.

H ~ h. The entry seems slightly wrong.* Klatt only mentions a single log-in of "90sAREover", and mentions three log-ins by Warman (two as "lucy" and one as "guest"). (Cf. his summary at p. 1648, where he says that the two accounts logged in "once or twice".) But even if this were correct, it is not helpful. All internet sites attract one-time visitors who don't return. (By way of comparison -- 90% of my traffic at this blog are first-time visitors, and another 8% visit 2-5 times; less than 2% return more than 5 times.) Why would this suggest they are identical?

*Update. Klatt's testimony is less clear than his affidavit (here), which shows that both lucy and 90sAREover did in fact log-in only twice.

I ~ i. The respective usage times differ by a factor of ten. That is hardly an argument for regarding them to be identical.

J ~ j. When they registered both "lucy" and "90sAREover" filled none of the optional boxes. The question is how typical this is. Personally, I never fill in more than I have to, and I suspect that is a common approach to message boards.

K ~ k. "Created for a single purpose" states more than can be known -- the "purpose" (or intention) of one or both of these accounts may have been either grander or more modest than came to pass. Indeed, given that the "lucy" account was never used to post anything at Freedomsite, it's difficult not to assume that its original purpose was something different than what it was actually used for, given that it was never used.

L ~ l. I assume that the dates here are offered for context, not as an argument in favour of identification.

Updated several times for clarity.

90sAREover's browser: RealOne with customized Explorer

Commenter "freemarkets" discovered an important fact when he pointed out (here) that the notorious Cools post was made by someone using RealPlayer, which has a built-in browser.  As it turns out, it was RealOne Player, version 2 (see here). Here is a screen shot in which I use this version of RealOne Player (downloaded from oldversion.com) to go the site useragentstring.com:

The tell-tale sign is the string "R1 1.3" at the end, which signals this specific version of the RealOne player.  

Another noteworthy fact, however, is "MSIE 6.0" earlier in the sting. This stands for Microsoft Internet Explorer 6.0. Explorer wasn't open when I took this screen capture, but these details are reported anyway. Why? As it turns out, although the RealOne browser seems perfectly adequate as a browser, it was not completely independent technologically, but depended on Internet Explorer for some of its functionality. In Jan. 2002 someone had complained to Real, Inc., that his RealOne browser wouldn't work after he had removed Internet Explorer from his system. Here is the answer he received from Real (which he then shared with the world through google groups):

"Please note that it is necessary to install MSIE 5.0x (Internet Explorer) or higher versions of the browser installed on your system for the RealOne Player to function properly. RealOne Player uses some components of Internet Explorer browser. I suggest you install Internet Explorer browser on your computer, then install RealOne Player again on your computer to resolve the problem. Please visit the Microsoft's web site (http://www.microsoft.com) to download the latest version of the MSIE browser."

So, Explorer is needed to run RealOne.

It is probably for this reason that whenever RealOne visited a site, its user agent communicated not only its own details in its user agent string, but also details about the version of Internet Explorer on the host computer. A long list of user agent strings in which we find some version of the RealOne Player can be at botsvsbrowsers.com.

But more can be said about this. As we saw, 90sAREover's user-agent string read something like this: "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Rogers Hi-Speed Internet; (R1 1.3))". Just as above, "R1 1.3" refers to the RealOne Player, and MSIE 6.0 refers to the browser that the RealOne used to help it surf the net. But this wasn't just any version of that browser, but one customized by Rogers and distributed to its customers: hence the "Rogers Hi-Speed Internet" tag (here).

The user agent string from Warman's visit shows that he had no such set-up: his version of Explorer was un-customized, and as I've shown (here), this argues against identifying him as the 90sAREover poster.

experimenting with RealOne and Explorer's customization options

When RealPlayer is used as a browser, the user-agent string left behind reveals not only the version that is used to visit a site, but also some details about the version of Internet Explorer that is resident on the surfer's computer.

We have already seen (here) that Microsoft allows customization of user-agent strings. I customized a version of Explorer by inserting "Buckets", surfed over to useragentstring.com, and took this screen shot:

Then I exited Explorer, rebooted, and visited the same site using the RealOne Player.  Here is the screen shot:

Note that the change to the user agent string that I had made for Explorer (inserting 'Buckets') is still there, even though I'm using RealOne, not Explorer.

The presence of "Rogers Hi-speed Internet" in the user agent of 90sAREover, therefore, requires that his browser was one of those customized by Rogers, even if he wasn't using it to directly visit freedomsite.org.

More on R1 1.3 as the user agent string in the Cools post of 90sAREover

We have already seen that the user agent string of 90sAREover (who posted a nasty racist screed against Anne Cools on Sept. 5, 2003) was "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Rogers Hi-Speed Internet; (R1 1.3))".

The reference to Rogers here is to a special version of Rogers' browser distributed to its subscribers.

In a user agent string, R1 1.3, refers to Real Player, and we can now state definitively what the specific version was: RealOne Player, version 2, and as you can see here (red arrow for the RealOne version; violet arrow for the user agent string):

The difficulty, as commentator freemarkets pointed out in the comments to an earlier thread, is that we can only get the RealOne Player to show up in the user agent string if it is actually the browser that is being used. That is, Real Player versions for Windows, at least since RealOne, have a browser built into them, so that you can surf the web using Real Player.

It seems that the only possible explanation is that the Cools poster 90sAREover was using RealOne as his browser. But an important question remains outstanding. How is it that "Rogers Hi-Speed Internet" found its way into the user agent string. Was there a special Rogers version of RealOne Player?