Klatt's Clunkers 8: Bernard Klatt comes close

Below and to the right is a long exerpt from Bernard Klatt's testimony of Feb. 8, 2007 (Richard Warman & the CHRC v. Marc Lemire, T1073/5405 vol. 9, pp. 1636-8) .

In it, Klatt, who had been summoned to testify as a technology expert, is discussing the phenomenon that we had identified in the last post: that is, in the logs of Sept. 5, 2003, the traffic associated with 90sAREover's visit sometimes appears as 66.185.84.204 and sometimes as 66.185.84.200. Indeed, in the three minutes following his notorious racist denunciation of Senator Anne Cools, the IP changes four times.

Klatt is unable to explain the phenomenon.

There are several troubling aspects of this. First, of course, is the fact that he he's not sure what is going on: one expects better of an "expert". Second, it is troubling that the shifting IPs (which is material to the identification that he's making) is a matter that was raised not by Klatt himself, but the Chair of Tribunal. He makes no allusion to it at all in his discussion of the Cools' post in his affidavit (paragraphs 34-41, here). His answer to the chair, however, reveals that this is something that he'd noticed before: "That one is interesting" (line 21 of p. 1636). Not interesting enough, apparently, that he should actually explain it in his testimony.

Most importantly, however, is that is a bit troubling how close Klatt came to the truth. At p. 1636, lines 22-23, for example, he recognizes that 66.185.84.200 might be a web-cache, which he repeats at p. 1638, lines 3-4, as "very likely". The correct answer, as we have seen, is that both these IPs are web-caches and that the shifting from one to the other is part of Rogers' practice of load balancing.  And the idea that he might suppose that 66.185.84.200 (which had a hostname of wc04.mtnk.rnc.net.cable.rogers.com) was a web-cache, but not guess that 66.185.84.204 (with hostname wc09.mtnk.rnc.net.cable.rogers.com) was also one is a bit baffling. He was so close!

One expects better than "close", however, from an expert. We want them to be right. Mistakes in expert testimony can ruin lives. In this case, the erroneous testimony has damaged the reputation of an innocent man and exposed those who have repeated Klatt's errors to financial harm.


Other posts in the Klatt's clunkers series:

• Klatt's clunkers 1: Bernard Klatt given an award for tendentious testimony
  
• Klatt's clunkers 2: Bernard Klatt wrong on IE 6 date
• Klatt's clunkers 3: Bernard Klatt misreads the user agent string
• Klatt's clunkers 4: Bernard Klatt mistakes the combined log format for the common log format
• Klatt's clunkers 5: Bernard Klatt doesn't recognize RealPlayer browser
• Klatt's clunkers 6: Bernard Klatt on Rogers service in BC
• Klatt's Clunkers 7: Bernard Klatt doesn't understand Rogers web caching
• Klatt's Clunkers 8: Bernard Klatt comes close on proxies
• Klatt's Clunkers: why Bernard Klatt should be disbelieved (summary thread)

Load-balancing in action: did Richard Warman and 90sAREover really have the same IP?

Below and to the right is an excerpt from the Freedomsite's logs, truncated on three sides to make it easier to read. (You can see an untruncated version here on p. 35.)

The top two entries (both with "POST") are related to the notorious Cools message: the first confirming the spelling of "nigger"; the second submitting the nasty message that is at the center of the present controversy. The bottom-most is the last entry for 90sAREover in the logs (or at least those that have been released).

The point to note is that 90sAREover did not have just one IP during his brief visit to the freedomsite message board, but two: 66.185.84.204 and 66.185.84.200, and in the space of three minutes he switches from one to the other at least four times.

I say "at least" four times here because it is important to remember that we are only seeing a portion of what was going on. These two IPs are both Roger's web-caching proxies -- their host names are (respectively) wc09- and wc04.mtnk.rnc.net.cable.rogers.com, showing the tell-tale wc (which abbreviates web-cache) -- and since they are caches there will have been traffic between them and 90sAREover's computer that is not forwarded to the visited site.

What does all this mean? The switching between these two web-caches is another example of load-balancing, a practice by which Rogers shifted traffic among its 42 web-caches to deliver the most efficient service to its subscribers. An attempt to sort out how all of this worked resulted in the string ball to the right (explained here), which traced shifts between proxy-IPs that can be identified in the years 2002-5. Each string in the ball is an example of a load-balancing shift.

This load-balancing provides another wrinkle in argument of identifying 90sAREover. It seems that Rogers' subscribers had what might be called a "home" proxy, which functioned as their default web-cache, and that they would rerouted to a another proxy only when needed, especially (one assumes) at peak hours. But when he made the racist Cools-post, was 90sAREover's "home" proxy 66.185.84.204 and 66.185.84.200 a temporary IP to which he was shifted through load-balancing? Or was his home proxy 66.185.84.200? Or might his "home" have been one of the other web-caches in the series, with the traffic rerouted into 200 and 204?

Klatt's Clunkers 7: Bernard Klatt doesn't understand Rogers web caching

As we have seen, one of the key pieces of evidence in the Cools-post debate has been the IP 66.185.84.204 (which has the host name wc09.mtnk.rnc.net.cable.rogers.com). As I have shown at length in this post, the wc stands for web-cache. A web-caching proxy server keeps copies of web-pages in its storage in order to save bandwidth. This practice, now in Roger's past, is important for the present controversy for two reasons. First, Rogers proxies hid the IPs of the individual computers that they are proxying for, and it was the proxy's IP that was logged at the websites visited. Second, because the proxy is performing this function for tens of thousands of Rogers subscribers at a time, the IP by itself is of little help in tying any one of them to any specific post.

In his testimony, however, Bernard Klatt seems to have missed this important fact. Indeed, when the topic of Rogers web-caching arose, he pleaded ignorance.

Since the IP in question was a such a proxy, and in fact the single most relevant fact to the case, Klatt's admission has the effect of rendering his testimony useless.

Other posts in the Klatt's clunkers series:

• Klatt's clunkers 1: Bernard Klatt given an award for tendentious testimony
  
• Klatt's clunkers 2: Bernard Klatt wrong on IE 6 date
• Klatt's clunkers 3: Bernard Klatt misreads the user agent string
• Klatt's clunkers 4: Bernard Klatt mistakes the combined log format for the common log format
• Klatt's clunkers 5: Bernard Klatt doesn't recognize RealPlayer browser
• Klatt's clunkers 6: Bernard Klatt on Rogers service in BC
• Klatt's Clunkers 7: Bernard Klatt doesn't understand Rogers web caching
• Klatt's Clunkers 8: Bernard Klatt comes close on proxies
• Klatt's Clunkers: why Bernard Klatt should be disbelieved (summary thread)

What to look for when recruiting an expert witness

This is a bad sign:

(For the context, see here and here.)

Load-balancing at Rogers

There is an interesting article in the networking magazine Network Computing of 1999 about Rogers' plans for the next few years. Concerned about the challenges that expected increases in traffic would bring, together with the expectation that e-commerce would become more important (making it all the more important that the traffic got through expeditiously), Rogers developed load-balancing technology to prevent server overload.  Some selected quotes:

Ten Alteon Networks ACEswitch 180 server switches balance the traffic flows among Rogers' five Web and cache servers, as well as its VPN and firewall servers. The Gigabit Ethernet server switches redirect traffic when one server goes down or gets jammed with HTTP traffic. The switches also route and handle the packet filtering for Rogers' firewalls, and all of Rogers' servers are connected via Gigabit Ethernet.

………

When a user requests Internet access, the switch directs that request to a proxy server. "It's load-sharing among cache proxies, and if they all aren't available, it then redirects the traffic to the Internet," rather than to the proxy server, Howell says.

This seems to be the explanation for the shifting IPs that we noticed earlier. Those shifts are taking place among various proxy servers.