Load-balancing in action: did Richard Warman and 90sAREover really have the same IP?

Below and to the right is an excerpt from the Freedomsite's logs, truncated on three sides to make it easier to read. (You can see an untruncated version here on p. 35.)

The top two entries (both with "POST") are related to the notorious Cools message: the first confirming the spelling of "nigger"; the second submitting the nasty message that is at the center of the present controversy. The bottom-most is the last entry for 90sAREover in the logs (or at least those that have been released).

The point to note is that 90sAREover did not have just one IP during his brief visit to the freedomsite message board, but two: 66.185.84.204 and 66.185.84.200, and in the space of three minutes he switches from one to the other at least four times.

I say "at least" four times here because it is important to remember that we are only seeing a portion of what was going on. These two IPs are both Roger's web-caching proxies -- their host names are (respectively) wc09- and wc04.mtnk.rnc.net.cable.rogers.com, showing the tell-tale wc (which abbreviates web-cache) -- and since they are caches there will have been traffic between them and 90sAREover's computer that is not forwarded to the visited site.

What does all this mean? The switching between these two web-caches is another example of load-balancing, a practice by which Rogers shifted traffic among its 42 web-caches to deliver the most efficient service to its subscribers. An attempt to sort out how all of this worked resulted in the string ball to the right (explained here), which traced shifts between proxy-IPs that can be identified in the years 2002-5. Each string in the ball is an example of a load-balancing shift.

This load-balancing provides another wrinkle in argument of identifying 90sAREover. It seems that Rogers' subscribers had what might be called a "home" proxy, which functioned as their default web-cache, and that they would rerouted to a another proxy only when needed, especially (one assumes) at peak hours. But when he made the racist Cools-post, was 90sAREover's "home" proxy 66.185.84.204 and 66.185.84.200 a temporary IP to which he was shifted through load-balancing? Or was his home proxy 66.185.84.200? Or might his "home" have been one of the other web-caches in the series, with the traffic rerouted into 200 and 204?

One of these things is not like the others: why Warman is innocent

Much of Canada's right-wing blogosphere is aflutter with the idea that human rights lawyer Richard Warman might have posted a racist comment about Canadian senator Anne Cools on freedomsite.com, a white-rights forum. The evidence? Freedomsite's logs preserve data from some visits of "lucy" (a handle Warman is known to have used) and from 90sAREover, who had posted a hasty racist screed against black senator Anne Cools. According to freedomsite's expert witness Bernard Klatt, both the IP and the user-agent data were identical for "lucy" and for "90sAREover". Until now we have had to take Klatt's word on this.

Recently these logs have become available for scrutiny (they were included in Klatt's affidavit, which I've recently posted here). The logs make it clear, however, that there are two important differences between the computer used during Warman's visits and those of 90sAREover (the Cools poster); both of these differences are exculpatory of Warman. Here is the evidence in reverse chronological order (compare what is underlined in red).

The log of Warman's visit from Nov. 23, 2003 (p. 33 of Klatt's affidavit):

The log of Warman's visit from Nov. 15, 2003 (p. 30 of Klatt's affidavit, here):

The log of Warman's visit from Nov. 11, 2003 (p. 29 of Klatt's affidavit, here):

The log of Warman's visit from Oct. 15, 2003 (widely cited, e.g. here):

All of Warman's visits from Oct. 15 — Nov. 23, 2003, show the same IP (66.185.84.204 = wc09.mtnk.rnc.net.cable.rogers.com; see the blue underline) and the same operating system and browser (MSIE 6.0; Windows 98 — see the red underline — which points to a generic version of Explorer 6.0 and Windows 98).

Here is the log of 90sAREover of Sept. 5, 2003 (p. 34 of Klatt's affidavit, here). Here the browser details are different (again underlined in red:

Although both Warman and 90sAREover have the same IP and operating system, 90sAREover's user agent string has two key differences from Warman's computer-setup. First, 90sAREover had a customized version of Explorer 6.0 installed, which had been supplied by Rogers (as is signaled by the phrase "Rogers Hi-Speed Internet"); second, he did not actually use IE 6.0 to post, but instead was using RealOne Player version 2 (hence "R1 1.3": see here). None of Warman's log entries, by contrast, show any sign of RealOne, and for all four of his visits his browser was recorded as a generic version of Explorer, not a Rogers one.

These differences are important and exculpatory and overthrow the technical argument for identifying Warman as "90sAREover", which (given that the IP used, 66.185.84.204, could have been anyone of millions of Rogers customers) has now collapsed. The computer that posted the Cools post was set up differently than the one Warman was using three months later.

(Re-written for clarity and context.)

Klatt affidavit in Richard Warman & CHRC v. Lemire

I've been wondering aloud for the past few weeks (here and here) why it is that although Warman's critics have produced scores and scores of documents (see here, for example), the actual evidence that he was the Cools poster were nowhere to be found. Logs were produced to show that Warman visited the site without logging in on Oct. 15, 2003, and these have been cited as some kind of proof (here and here). But where were the logs of the notorious Cools post? Why was some evidence being fed into the blogosphere, but the most important evidence not?

Someone pointed out to me that the logs were part of the original Klatt affidavit, which I've managed to acquire. So that you can see the whole thing, I've uploaded into scribd.com and embedded it below. There are several serious mistakes in Klatt's testimony that I have already identified, and I hope to return to these soon. For now, however, the logs of the notorious "Cools" post is on p. 35, and these show that the case against Warman is fatally flawed: 90sAREover had a different browser installed than what we know Warman used for all of his posts in October and November. I'll clarify this later today.

Read this doc on Scribd: Klatt affidavit - 22 Aug 2006

Revisiting the lucy-90sAREover identification

Those who have been following the controversy about the Cools post will probably know the following chart. It apparently imitates one that Bernard Klatt presented at the CHRC hearing -- Klatt refers to such a chart during his testimony -- but this version (with my mark-up in red for point-by-point reference) was posted freedominion.ca on January 20 by "Entropy Squared". It is now part of Warman's statement of claim.

The chart was designed to make a case that Richard Warman was responsible for the posting of a racist denunciation of Senator Anne Cools on Sept. 5, 2003. The evidence argues against that conclusion. (Again, what is in red is my mark-up.)

A ~ a and B ~ b. Warman registered at Freedomsite.org using the pseudonym "lucy" and the IP 66.185.84.204, according to Klatt's testimony. Klatt also testified that 90sAREover (who wrote the notorious Cools post) had the same IP. I have shown here, however, that this IP is one of Rogers' web caching servers and could be used by any Rogers' subscriber -- indeed, the post could have been made by anyone with access to a Rogers-connected computer. This could be any one of millions (here).

• Update. Actually, the IP of "90sAREover" is less clear in the logs than Klatt's testimony implies: in the few minutes that 90sAREover spent at the site, his IP can be seen shifting back and forth between 66.185.84.204 and 66.185.84.200 five or six times. See here.

C ~ c. According to Klatt's testimony, the user-agent string recorded in the logs freedomsite.org show that both "lucy" and "90sAREover" used Windows 98. According to W3schools.com, 12.1% of computers were using Windows 98 in Sept. 2003, when the Cools screed was posted. 12% of millions (see previous), however, is still hundreds of thousands.

D ≠ d. During his testimony at the CHRC, Klatt asserted that "lucy" and "90sAREover" used the same browers. This is wrong. Details of a visitor's browser are found in the user-agent string, which is recorded by many sites. To judge from Klatt's testimony (here)*, the user-agent string of 90sAREover included the phrases "Rogers Hi-Speed Internet; (R1 1.3))". The R1 1.3 means that 90sAREover was not using Explorer as his browser, but in fact was using RealOne Player (version 2). The reference to "Rogers Hi-Speed" shows that the Cools poster had a version of Explorer that had been customized by Rogers for its customers (here), even though he was not using it to visit this site. To judge from Klatt's testimony, none of Warman's own visits show either the RealOne Player or the Rogers-customization. This argues against identification. (One wonders why there are not copies of these logs floating around the internet -- is it because they exonerate Warman?)

*Update. Now that the site logs are available for inspection, rather than Klatt's testimony, it is clear that Richard Warman's browser set-up was different than 90sAREover's.

E ≠ e. The IDs and emails are different. This can hardly be used as an argument that they're run by the same individual. Indeed, those trying to argue that they are identical need to explain why Warman registered "lucy" in mid-November (an account that he never posted from) if he already had the 90sAREover accout. If he were "90sAREover", why wouldn't he simply have used that account rather than registering the new one?

F ~ f. It is a matter of public record that "lucyaubrack@yahoo.ca" was Warman. We don't know, however, whether the "rob_m_simpson@hotmail.com" account was anonymous or whether it reflects someone's actual name.

G ~ g. Both "lucy" and "90sAREover" found their way to freedomsite.org, which was relatively obscure. This no more requires that these two were identical, however, than it would for any of the site's others users, or indeed for visitors of other obscure sites (such as the one you are visiting now). In any case, it is worth noting that 90sAREover is member #1331 (affidavit, p. 10), and "lucy" (Warman's handle) is member #1379 (affidavit, p. 10), which implies that at least 50 new members joined up in this period. These surely aren't all the same individual.

H ~ h. The entry seems slightly wrong.* Klatt only mentions a single log-in of "90sAREover", and mentions three log-ins by Warman (two as "lucy" and one as "guest"). (Cf. his summary at p. 1648, where he says that the two accounts logged in "once or twice".) But even if this were correct, it is not helpful. All internet sites attract one-time visitors who don't return. (By way of comparison -- 90% of my traffic at this blog are first-time visitors, and another 8% visit 2-5 times; less than 2% return more than 5 times.) Why would this suggest they are identical?

*Update. Klatt's testimony is less clear than his affidavit (here), which shows that both lucy and 90sAREover did in fact log-in only twice.

I ~ i. The respective usage times differ by a factor of ten. That is hardly an argument for regarding them to be identical.

J ~ j. When they registered both "lucy" and "90sAREover" filled none of the optional boxes. The question is how typical this is. Personally, I never fill in more than I have to, and I suspect that is a common approach to message boards.

K ~ k. "Created for a single purpose" states more than can be known -- the "purpose" (or intention) of one or both of these accounts may have been either grander or more modest than came to pass. Indeed, given that the "lucy" account was never used to post anything at Freedomsite, it's difficult not to assume that its original purpose was something different than what it was actually used for, given that it was never used.

L ~ l. I assume that the dates here are offered for context, not as an argument in favour of identification.

Updated several times for clarity.

90sAREover's browser: RealOne with customized Explorer

Commenter "freemarkets" discovered an important fact when he pointed out (here) that the notorious Cools post was made by someone using RealPlayer, which has a built-in browser.  As it turns out, it was RealOne Player, version 2 (see here). Here is a screen shot in which I use this version of RealOne Player (downloaded from oldversion.com) to go the site useragentstring.com:

The tell-tale sign is the string "R1 1.3" at the end, which signals this specific version of the RealOne player.  

Another noteworthy fact, however, is "MSIE 6.0" earlier in the sting. This stands for Microsoft Internet Explorer 6.0. Explorer wasn't open when I took this screen capture, but these details are reported anyway. Why? As it turns out, although the RealOne browser seems perfectly adequate as a browser, it was not completely independent technologically, but depended on Internet Explorer for some of its functionality. In Jan. 2002 someone had complained to Real, Inc., that his RealOne browser wouldn't work after he had removed Internet Explorer from his system. Here is the answer he received from Real (which he then shared with the world through google groups):

"Please note that it is necessary to install MSIE 5.0x (Internet Explorer) or higher versions of the browser installed on your system for the RealOne Player to function properly. RealOne Player uses some components of Internet Explorer browser. I suggest you install Internet Explorer browser on your computer, then install RealOne Player again on your computer to resolve the problem. Please visit the Microsoft's web site (http://www.microsoft.com) to download the latest version of the MSIE browser."

So, Explorer is needed to run RealOne.

It is probably for this reason that whenever RealOne visited a site, its user agent communicated not only its own details in its user agent string, but also details about the version of Internet Explorer on the host computer. A long list of user agent strings in which we find some version of the RealOne Player can be at botsvsbrowsers.com.

But more can be said about this. As we saw, 90sAREover's user-agent string read something like this: "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Rogers Hi-Speed Internet; (R1 1.3))". Just as above, "R1 1.3" refers to the RealOne Player, and MSIE 6.0 refers to the browser that the RealOne used to help it surf the net. But this wasn't just any version of that browser, but one customized by Rogers and distributed to its customers: hence the "Rogers Hi-Speed Internet" tag (here).

The user agent string from Warman's visit shows that he had no such set-up: his version of Explorer was un-customized, and as I've shown (here), this argues against identifying him as the 90sAREover poster.

More on R1 1.3 as the user agent string in the Cools post of 90sAREover

We have already seen that the user agent string of 90sAREover (who posted a nasty racist screed against Anne Cools on Sept. 5, 2003) was "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Rogers Hi-Speed Internet; (R1 1.3))".

The reference to Rogers here is to a special version of Rogers' browser distributed to its subscribers.

In a user agent string, R1 1.3, refers to Real Player, and we can now state definitively what the specific version was: RealOne Player, version 2, and as you can see here (red arrow for the RealOne version; violet arrow for the user agent string):

The difficulty, as commentator freemarkets pointed out in the comments to an earlier thread, is that we can only get the RealOne Player to show up in the user agent string if it is actually the browser that is being used. That is, Real Player versions for Windows, at least since RealOne, have a browser built into them, so that you can surf the web using Real Player.

It seems that the only possible explanation is that the Cools poster 90sAREover was using RealOne as his browser. But an important question remains outstanding. How is it that "Rogers Hi-Speed Internet" found its way into the user agent string. Was there a special Rogers version of RealOne Player?

Where are all the logs?

I have been posting for several weeks on the question of whether, as some have argued, "lucy" (a pseudonym that we know belonged to Richard Warman) was the same person as "90sAREover" (who posted a nasty racist denunciation of Anne Cools), as has been alleged by his enemies. I've already shown that the IP being used, 66.185.84.204, is a web-caching proxy that might be almost anyone in Ontario. And in the last post I pointed out that Klatt's testimony in Warman v. Lemire (see here) about 90sAREover's user agent string on Sept. 5, 2003, seems to imply that it was performed on a different computer than the one that Warman used on October 15.

But all this raises a question that should be uncomfortable for those who have argued that Warman is the Cools poster. Why is it that these logs have not been released? Is it because they are inconsistent with the claim that Warman had written this?