Security Manager's Journal: When is a patch not really a patch?
Answer: When the patches are dutifully installed but the servers aren't rebooted -- for more than two years. I made a disconcerting discovery last month when Microsoft released a security patch outside of its regular monthly routine. How disconcerting was it? I'm talking about the discovery that a lot of the patching we've done to our more than 450 servers has never gone into effect.
Let me give you the background.
Each quarter, I present metrics to our CIO, including the percentage of servers that are compliant with the latest recommended critical security patches. Our Windows desktop and server team meets once a month to review the latest Microsoft patches and decide which ones we should apply based upon criteria we use to classify the threats. Patches we choose to apply are pushed to our servers and desktops, and then I check our Microsoft Systems Management Server to gather the information for the CIO.
I'd always assumed that when the SMS reported that a particular server had been patched, I could run with that information. Nope; there's a gotcha.
If you don't reboot a Windows server after a patch is applied, the patch doesn't take effect, but SMS doesn't notice that failure to reboot. This insistence on rebooting is one of the things I dislike about Windows. In the Unix world, all that's usually required is that a particular process be restarted.
When that so-called out-of-band patch came up, I attended a meeting to discuss whether it would be necessary to install it on all of our servers and 8,000 desktops. From what I had read ahead of the meeting, I was convinced that it was absolutely necessary. The patch fixes a vulnerability in the "server service," which lets Windows communicate with network devices. A remote procedure call could cause a Windows system to execute arbitrary code, and reports were already circulating that exploit code and a worm had been developed. In other words, this was one vulnerability we couldn't afford to ignore.
Never Assume...
It was during that meeting that I made my disconcerting discovery. The IT managers were freaking out that they would have to reboot over 450 servers. My first reaction to their concern was confusion. Making assumptions can get you into trouble, and my assumption that the servers were routinely rebooted after patches were installed certainly fit the bill. I found out, in fact, that many of our servers hadn't been rebooted in over two years. They had more than 24 months' worth of patches installed on them that were doing us no good.
I also came to find out that there was no comprehensive list of servers with asset-identification data, such as the applications running on each server, points of contact, and whether the server was for production or development. As a matter of fact, about a quarter of the servers were in an unknown state, meaning no one knew what their purpose was. Scary. This is what happens when you don't have a sound configuration and asset management program in place. We spent the next several days and a large chunk of the IT organization's time trying to figure this out, and still we were unable to identify about 15% of the servers.
As for that out-of-band patch, we decided to push it to the servers and then reboot all of them on a Friday evening. As luck would have it, there were no major issues, and all the servers came back online, patched and apparently running smoothly. However, there may be some fallout over the next few weeks as applications that aren't frequently used start to show the effects of this big patch implementation.
Oh, and I'm happy to report that the new version of SMS, called System Center Configuration Manager, can report on whether a server is awaiting reboot.
Reproduced from an article published by ComputerWorld
© ComputerWorld
The original article can be viewed here:
http://www.computerworld.com/action/article.do?command=viewArticleBasic&tax...
Permalink Bookmark Digg this story
