Windows XP SP2 is big step forward in security--but it can break things

By John McCormick | More from John McCormick | published: 6/7/04
Category: System Intrusions | Audience: NetAdmin
Rating: 4.4 (out of 5) Rate it Comments: 46 | 0 NEW | View all
Update - 8 days ago: Stay on top of the latest XP tips and tricks with our free Windows XP e-newsletter, delivered weekly. Automatically sign up today!

Learn about the plethora of security enhancements that Microsoft has included in Windows XP Service Pack 2, as well as how these security features could impair the functionality of some applications.

For more Windows XP Professional solutions, check out TechRepublic's Windows XP Professional Resource Guide. With this book and CD-ROM set you'll master the total XP package: from learning new features and functionality to upgrades and installation, networking, administration and security issues.

The release of Service Pack 2 for Windows XP will mark a milestone in the life of this operating system. Microsoft is pulling out all the stops to improve security. So much so, in fact, that it will cause many problems—because SP2 will de-emphasize backward compatibility with legacy systems and code for the sake of security. Administrators need to know in advance just what SP2 will mean for the Windows XP systems on their networks.Based on Service Pack 2 RC1

The information in this article is based on Windows XP Service Pack 2, Release Candidate 1. There could be a few minor changes to the software in the final release, but nearly everything you read here will still apply.

Details
Windows XP SP2 will dramatically improve the default security configuration for XP in nearly every area from e-mail to Web browsing to increased protection against the ubiquitous buffer overrun. But, all of these security improvements won’t come without some considerable pain. ZDNetUK reports that Microsoft admits that as many as one application in 10 will experience problems due to the upgrade (I consider that a conservative estimate).

Here are some of the most important security changes that are part of XP SP2:

* The Internet Connection Firewall is now enabled by default, which should improve security for SOHO users. However, in a corporate environment it could cause problems for users trying to connect to network resources. The firewall will also now activate much earlier in the boot cycle, even before the network stack is enabled. On shutdown, it will now remain active until after the stack is disabled.
* The Messenger service is now disabled by default.
* A pop-up ad blocker has been turned on by default.
* A unified security application called the Windows Security Center has been added (for more information on this feature, see this News.com article). It is supposed to bring all of the most basic security configuration information into one easy-to-manage place that will show whether your firewall is enabled, if your antivirus software is working, and if you have the latest software updates installed.
* NX support is added to Windows XP. NX (no execute) will allow NX-enabled CPUs to mark certain areas of memory as non-executable; that is, any code pushed into those areas (perhaps by malware such as Blaster or other viruses) will just sit there, unable to run and therefore will be rendered harmless. This will harden the OS against the notorious buffer overrun threats. NX is currently only supported for AMD’s K8 and Intel’s Itanium processors, but 32- and 64-bit support for this important security feature is expected in most future processor releases.
* DCOM (the Distributed Component Object Model) gets a new set of restrictions in the form of an access control list for nearly every action of any COM server. There will also be a more detailed set of COM permissions, which will allow administrators to fine-tune COM permission policies.
* There is improved port management. It will no longer be up to the application to close ports after it is finished. Before, if a developer left out the closing routine or the application crashed, a port could remain open and leave XP open to attack. SP2 encourages port management with an application white list that only a user with administrator privileges can alter. Placing an application (such as a peer-to-peer program) on the white list causes ports to be managed automatically. Such applications can also now be run as a regular user rather than needing local administrator privileges to open ports in ICF.
* New RPC restrictions help tighten communications. The XP SP2 changes in this area let administrators fine-tune RPC services. This granular control over RPC will allow you to specify that a port be used for RPC even if the application is not on the white list. There are a lot of changes for RPC, including a new RestrictRemoteClients registry key that by default blocks most, but not all, remote anonymous access to RPC interfaces on the system. The RPC interface restriction will require an RPC caller to perform authentication, which makes it much more difficult to attack an interface, and helps mitigate against Trojan attacks.

Windows XP SP2 is big step forward in security--but it can break things
Part 2

Potential problems

The NX protection mentioned above is an excellent example of something that is definitely a powerful improvement from the security standpoint. However, NX has already been reported to have caused considerable problems (at least in the 64-bit version). The biggest problem will come for applications that use just-in-time code creation. On the other hand, the .NET Framework common language runtime code already supports NX as implemented in SP2.

RPC changes are the most likely to wreak havoc with existing applications. In the pre-SP2 Windows XP implementation, there are literally scores of RPC-based services running, all of which provide a window for attack. That changes dramatically with SP2.

Because of the change in port management, if an application needs to open ports but doesn’t use stateful filtering, administrators installing it need to place the program on the white list. With the built-in firewall enabled by default, IPv4-application inbound connections for audio and video, such as for MSN or Windows Messenger, need to have their port opening and closing managed automatically. Inbound services connections (IPv4) will require some changes to configuration and/or code. Services that listen on fixed ports should ask users if the service should be permitted to open the port in ICF and, if so, the service should use the INetFwV4OpenPort API to alter ICF rules.

Another problem is the fact that Microsoft won’t be offering this service patch to those who hold pirated copies of Windows XP, which is reasonable enough, but there are a lot of illegal copies out there, especially in the Far East where a lot of worms get a quick foothold in the Internet. SP2 will apparently check Product IDs looking for known pirated copies and will not install on systems with bad Product IDs. This is understandable, but will reduce the overall effectiveness of the security upgrade.

A lot of the potential problems posed by SP2 are beyond the control of administrators. Some programming code for custom applications will have to be rewritten, but at least now you know what to look for when problems come up, rather than deploying XP SP2 and finding out that it breaks your most important line-of-business application.
Further reading

This report can only scratch the surface of such a major overhaul to an operating system. For more information see:

* XP SP2 Preview
* XP SP2 RC1 Fact Sheet
* XP SP2 RC1 Release Notes
* XP SP2 Developer Notes

Final word

I expect to hear screams of pain as people deploy SP2 and discover that legacy applications no longer work, but those are probably the same people who complain so loudly (and legitimately) that Microsoft doesn’t deploy secure systems.

Nearly every security expert knew that, at some point, Microsoft would be forced to bite the bullet and take a big compatibility hit in order to solidify operating system soft spots—many of which are due to legacy code support. Plus, the XP SP2 changes will force developers to produce more secure applications and not just take advantage of a permissive Windows OS to write code that doesn't pay attention to security.

Of course, I would never recommend that anyone deploy such a major upgrade widely the day it hits the street. You should install SP2 on a testing network (or at least a single testing system) as soon as possible, and begin compatibility testing for your specific applications.

Those of you who have the budgetary luxury of being able to conduct even more extensive testing and want to get a leg up on evaluating XP SP2 even before final release should check out the Technical Preview Program, which makes SP2 RC1 generally available for testing by IT professionals (not just those on the beta list). The initial download, which doesn’t include any support other than some Microsoft-sponsored newsgroups, requires Windows XP to be installed already. English and German versions of the update are now available and are about 270 MB in size.

As soon as you feel comfortable that Windows XP SP2 will not cause a significant interruption for users (or you have fixed the issues that would lead to a potential interruption), then you should deploy SP2 company-wide. It is an important upgrade that can only improve the security of your network.
Also watch for…

* There's one item to highlight this week. Silicon.com and other sources are reporting that Apple’s recent patch to fix a major threat in Mac OS X wasn’t completely successful, and that a highly dangerous problem still exists in the operating system. The threat is especially noteworthy because it is the first important vulnerability discovered in the Mac OS X operating system that was not due to a flaw in the underlying FreeBSD UNIX on which Apple based OS X. This problem lies in the part of the code created by Apple, and it appears that it is quite difficult to repair. This is the first real challenge to Apple, and it will be interesting to see how the company responds to this critical threat. Previous patches were simply carried over from the Linux/UNIX community. Apple is on its own this time.