Back in April of 2008, Microsoft pushed Office Genuine Advantage out to Customers' WSUS servers world-wide, though the tool was only supposed to be distributed in a targeted geographic area. This was acknowledged as a "mistake" by Microsoft. No method of removal was provided.

In November 2007, Microsoft renamed a product category in the head-end WSUS servers and broke the user interface on Customers' WSUS servers. This was rectified in a subsequent update. Quoting a note from the WSUS team: "We are also improving our publishing tools to make sure that issues like this are caught during the publishing process, before they impact customers." (It would seem that this relates only to catching this particular issue-- assuming we don't see it happen again.) If you were one of the unlucky Customers to receive the bad data, you were stuck performing a manual resolution procedure!

In October 2007, Windows Desktop Search was widely deployed to desktop computers, inadvertantly, by Customers using WSUS. Microsoft cited the "decision to re-use the same update package" as having "unintended consequences to our WSUS customers". No automated solution was provided to undo the damage done to potentially large numbers of computers. Windows Desktop Search did get a boost in installed base, though.

In September 2007, Microsoft caused the WSUS servers of Customers who opted to synchronize hardware driver updates to see approximately 4,000 new updates for ATI graphics cards. The WSUS team noted: "We are changing the publishing process for the future btw so that multiple HWIDs will be associated to one update in the future." Customers received metadata for the 3,982 seemingly duplicated updates were given instructions on manually rectifying the situation themselves.

In November 2006, Microsoft released Internet Explorer 7, Spanish locale, to all locales (not just Spanish). The error was confirmed by Microsoft and updated metadata was scheduled to be deployed. At the time, Microsoft's representative stated "We regret the inconvenience and confusion this issue may have caused WSUS customers. Thank you for your reports and enabling us to get this issue headed off so quickly." It is fortunate that so many Microsoft Customers work as unpaid regression and quality-assurance testers.

(I'm not even going into the months-long fiasco about "SVCHOST.EXE" hanging older PCs and the multiplicity of "fixes" that didn't actually resolve the issue proffered by Microsoft. That's probably more a beef with the "Windows Installer" people than with the WSUS people.)

After all of this, we now have a situation where bad data gets synchronized into Customers' WSUS databases causing unhandled errors in the server-side code called by client computers looking for updates. Beautiful.

So far, the only resolution I'm aware of involves a manual procedure performed by the Customers. This is also beautiful. I've already had the issue in at least one Customer site.

Is there any regression testing being done on patches deployed thru WSUS? Is there regression testing of the patch metadata being synchronized into Customers' WSUS databases? It sure doesn't look like it, on either front.

Why can't Microsoft take the time to provide automated fixes for the damage it creates automatically. It's not as if they can't write code to do things automatically.

It has the look and feel that a single disgruntled (or stupid) Microsoft employee could bring down a large portion of the desktop PCs and servers in the world. I won't even think about malicious third-parties gaining access to the server computers that serve updates out to privately owned WSUS servers throughout the world. Seemingly, if some catastrophe like this did happen, Microsoft would release a procedure for their Customers to manually perform on each affected system. Whee!

Yet again, I'm embarrassed to have my Microsoft "certification" and to be associated with them in any way. Way to foster trust in IT, Microsoft!

I ran into an odd issue with a friend's network yesterday, and have decided that feeding it to the LazyWeb to chew on is a good idea.

For semi-relevant informational background, the subject network in this diatribe is a small private religious K-12 school w/ roughly 120 Windows XP Professional SP2-based PCs. They have a Windows Server 2003 Standard Edition R2 32-bit file server computer acting as an AD domain controller, a Windows Server 2003 Standard Edition R2 32-bit server computer acting as a replica domain controller, and a Windows Server 2003 Standard Edition R2 64-bit server computer running Exchange 2007. Everything was a clean install migrating away from Novell Netware last September, and all the client computers were reinstalled from fresh Windows XP installations at the time of the migration. The network infrastructure is all Cisco-based switched 10/100 Ethernet (w/ gigabit uplinks between switches) with no VLANs or QoS. I did most of the original setup, and things are sanely configured (clients pointed to internal DNS servers running on domain controllers, IP addresses handed out via DHCP, etc). In general, everything has been humming along since I did the initial setup, and walking in to look at the issue I didn't expect that it was anything setup-related. (Because of bogus political reasons, I can't bill for work on this network anymore, but I became friends with the on-site "computer teacher" and I still stay in touch with him. It's frustrating, but I like the people and try to generally be helpful and nice... *smile*)

Okay, okay-- enough blathering on. The issue shakes out like this:

Last week, the client computers (most of them current on Microsoft updates as of April 20th or so) started hanging during common user activities-- mainly opening and closing Microsoft Office and Adobe CS3 applications and using Internet Explorer. Even Windows Explorer would hang, from time to time. If one would leave the computers sit in this "frozen" state, they would eventually "free up" and begin to work again. In cases where a hang occurred closing a program (such as WINWORD.EXE), the program might hang around in the process list for awhile and eventually disappear. You could open more copies of the program, and as you closed them, you would build up more "hung" copies in the process list.

My friend and I found that we were able, just by fiddling around with Microsoft Word, Adobe Illustrator, etc, to reliably generate failures in about 3 - 5 minutes of work. Strangely, though, we could only get failures to occur when logged-on as a user who did not have local "Administrator" rights.

Of all the users on the network, only one (1) user logs-on with a non-limited "Administrator" account (for frustrating reasons I won't go into). We checked with this user and found that she has seen no issues. This seems to jibe with our inability to reproduce the issue except when logged-on as a limited user.

Watching the hangs with Process Explorer, I was seeing several threads in the hanging programs stuck on calls to kernel32.dll's GetModuleFileNameA+0x1b4 export. I think this is related to the root-cause of the issue, but I don't have the right source code to debug this any further down into the stack. Anyway, I kept banging on Process Explorer for a bit, but then we moved on to think about other things that might've changed.

A major "changed" item that we discovered related to the Trend Micro OfficeScan product. The OfficeScan "32-bit Virus Scan Engine" was updated on 4/22/2008 to version 8.700.1004. My friend recalled that the problems being reported by users starting last Tuesday, and a quick review of the trouble ticketing system revealed that this was the case-- all the trouble reports started on Tuesday after the Trend Micro update.

I'd already gotten the feeling that the root cause was probably anti-virus related, simply because the issue was happening to such a variety of computers and in a variety of applications. The only commonality between the machines, aside from the operating system, was the anti-virus software. In an earlier test, we removed OfficeScan from a machine on which we had been able to reproduce the issues and tried for 30 minutes to reproduce the issues without success. We allowed Group Policy to reinstall OfficeScan and reproduced the issue again within 5 minutes.

I performed a "rollback" to OfficeScan virus scan engine version 8.550.1001 on our test client computer (via the OfficeScan console). We verified that the client reported the older scan engine, bounced the machine, and spent 30 mintues attempting to reproduce the issue. We could not reproduce the issue with scan engine 8.550.1001. We rolled the engine forward to 8.700.1004 again and were able to reproduce our issue.

For now, we've initiated "rollbacks" on all the client computers that are "online", and my friend will watch tomorrow and rollback any other clients that don't pick up the rollback request automatically. I don't like not being current on updates to things like anti-virus software, but I think it's a necessary evil in this case, and because it's only the scan engine and not the virus definitions, we are probably not opening ourselves up to undue risk.

The only thing I found on the 'net thusfar was a vague posting, and it's too vague to really get anything out of.

How about it, Lazyweb? Any similar situations happening out there?

For reasons that I cannot identify, I decided to chart the number of results returned by Gooogle for various spellings of the word "poop".

I also made a view into the "long tail" of "poop" (showing the odd spike at 44 O's.)

I have absolutely no explanation for doing this.

Pooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooop.

If you'd like, I've also made the raw data available for remixing.

I just got back from the This American Life live simulcast in my local movie theatre. It was a really enjoyable show, and we finally got to see what Tory Malatia looks like! (The moving camera was somewhat disquieting, at first. There was major, major jib action going on...)

I really hope that the live show makes it onto the second season DVDs as an "extra". I hope the pre-show animation (a parody on the standard faire movie theatre pre-show trivia reels) makes it on the DVD as well.

Anyway, it was a great show. I hope more people get to see it. It was a unique experience, and a lot of fun.

"Pushing crap
On the stack.
Pop it back
Into registers, memory.
Hey-- I'm ready to code."

Been in my head for years, never committed it to bits 'til now.

I think I'm safe to go ahead and write this one, what w/ only one "working day" left in 2007. We'll see...

So, a user (we'll call her "Shelly") calls me indicating that her vertical-market time and billing accounting application is not spell checking data she enters. Further, Shelly states, she spent over an hour on the phone with the application manufacturer's technical support, and they've decided that the registry on her Windows XP Professional-based PC is "corrupt". Other users in the office don't seem to have any problem, but she can enter violently misspelled words and see no squiggly red lines warning her of misspellings.

I choked back my initial urge to comment on the use of the word "corrupt" ("What, has it been taking bribes or something?!?"), since I firmly believe that the words "corrupt" and "registry" coming out of a technical support representative's mouth together are really code for "I don't know what's wrong but I don't want this to be my problem anymore." I also choked back my urge to comment on the obscene yearly "maintenance" fee required by the software manufacturer that, apparently, entitles their Customers to receive slipshod, finger-pointing "support".

I agreed that I'd fire up a VPN connection, hop onto her PC with a remote desktop control application, and "ride along" to see what was happening.

I got connected to Shelly's PC and she demonstrated, in her characteristic ALL CAPS TYPING STYLE, the lack of spell checking. I've always cringed at the ALL CAPS-ness of her typing, but I know that her ways cannot be changed. She's been doing this since her office got time and billing accounting software back in the late 1980s.

"Did tech support have you logon to another PC where the spell checker works fine to see if the problem 'follows' your user settings," I asked. I was betting the spell-check component has both machine and user registry settings, and given that Shelly's office uses roaming profiles, a mal-adjusted user setting should 'follow' her, whereas a machine setting should stay in place.

"Uhh, no", she said. That's a strike against software manufacturer technical support, to me. I suppose they couldn't know that we were or weren't using roaming profiles, but even w/o roaming user profiles, different users on a Windows NT-derived operating system have different user registries.

I think it's reasonable to see if you can isolate an issue to being a "per-machine" or "per-user" issue at level 1. Of course, since they wrote the software to begin with, you'd think they'd already know where their settings were stored. To be honest, though, I'd guess that even their development staff don't have any idea where these supposedly "corrupt" settings are stored. My bet would be that they don't even have a way to talk to the people who wrote the spell checker, seeing as how it's a licensed third-party library.

I convinced Shelly that we should make sure the problem wasn't with her user settings. "But they said my registry is corrupt! My registry! Corrupt!" Oh, the horror. Luckily I've got a pretty good track record of solving problems for Shelly, and she let me proceed as I wanted.

I logged-on with my test user account and we attempted to reproduce the issue. My account received proper spell checking, unlike Shelly's. At least I knew that the underlying code was working fine, and likely this was a user registry issue.

I decided to head into the menus and have a look at the spell check options. Ultimately, I wanted to see where the settings are being saved in the registry. I figured I'd do my usual trick of taking a snapshot of the registry, changing a setting and closing the app, then taking a snapshot again and comparing the snapshots. (Sure, sure-- I could just run RegMon while I use the app, but I can do this faster and w/o downloading software.)

I never even made it to taking my "before" snapshot, though. When I looked at the options dialog, one (unchecked for my account) item jumped right out at me: "Ignore words in ALL CAPS".

"Umm, Shelly-- let's logon as you again and check one setting."

Sure enough, the setting "Ignore words in ALL CAPS" was checked in her settings. I read the setting's name out loud, and she replied: "Well, I set that because I type everything in all capital letters and I wanted it to..." She trailed off, then paused. "Oh," she said, "I understand. Sorry to have bothered you."

"No problem," I said, and disconnected from her PC. Nothing more could be said.

I suppose the only thing that could've made this better would've been if the spell checker had a setting to "Ignore misspelled words" and she'd enabled that.