Patching – Past and Present
As IT Administrators are becoming increasingly aware, one of the biggest challenges in organizations of all sizes is the issue of keeping up with all of the operating systems and application patches that are released on a regular basis. Most organizations will have some type of policy regarding the patching of their servers, but now what about the bigger challenge of patching all of those workstations? Bigger, only because there are so many more of them out there – there may be 50 to 100 workstations for every server in many larger organizations.
To patch, or not to patch? That is the question. Actually, this is only one facet of the question. How do we implement the patches? Who decides when and whether to apply patches? How do we keep track of which machines have which patches? These are but a few questions that can have IT Administrators ripping their hair out (which may explain the increasing popularity of shorter haircuts these days).
In days gone by, applying a workstation OS or application patch to the user population meant sending out a “SWAT team” of technicians to every PC in the organization, disrupting the workers while we took command of their keyboards, applied the software patches, rebooted, and then handed back the (hopefully) improved system back to its user. At least, we hope these are days gone by – you’re not still doing this, are you?
This might be workable if you only have 5 or so computers in a small office (or perhaps 1 or 2 computers at home). But even with as few as 10 or more computers, this manual process can quickly become very onerous. We could get away with it when operating system or application patches came out once every six months or so, but now – with as many as 15 patches per month, this gets crazy in a hurry.
So the “sneaker method” (visiting every workstation) of software patching is definitely out – that is SO 20th century. In the past 10 years, many software vendors, including Microsoft (by far the largest generator of software patches), have developed methods to automate this process of software patching.
A significant development in this arena was the institution of Windows Update – Microsoft dedicated a Web site, as well as operating system enhancements, to allow end users to download and apply Microsoft operating system patches somewhat automatically. Initially, this still required end user intervention – the user had to go to the Windows Update site, perform an online “Search for Updates”, and then select and install the recommended operating system patches. This may be fine for the individual home user or for SOHO users, but hardly efficient for larger organizations.
With Windows XP, Microsoft took this self-patching model to the next level, so that the individual Windows PC can, with no deliberate end-user involvement, automatically detect, download and install operating system patches. In this model, we allow Microsoft to determine that we need the patch, and we also let them deploy it for us.
Again, the Windows automatic-update capability probably works great for the home user, but do we want 500 computers in the office to each individually pull down XP Service Pack 2 (~300MB)? Or do we want to depend upon each individual end user to decide whether or not to apply these patches, and when? And then how in the world do we keep track of who is patched with what patches and who is not?
Also, do we want “big brother” (the software vendor) to decide for us which patches to deploy, and when? For example, is Internet Explorer 7 really a critical patch, or just a software upgrade to compete with Firefox? What if our Web applications no longer work with the new browser? I think most IT administrators would much rather have the control in their own hands to determine when and which patches to deploy.
Many software vendors have jumped into the patching arena, with client/server solutions which allow administrators to centrally select and deploy operating system and application patches to the end user PCs – often with no end-user intervention, or even awareness, in some cases, of the patching events.
While the nirvana of an IT Administrator may be an operating system that never needs patching, this may be the next best (realistic) thing. We have automation combined with control, and we have reporting capabilities to tell us how we are doing – with status reporting giving us everything from the patch status of individual computers, all the way to the collective patch status of the entire organization.
One of the first vendors (and reputedly one of the best) in this arena is the PatchLink Corporation, which (as seen by the name) specializes in managing and deploying OS and application patches. PatchLink employs a Web server running the PatchLink application to download and deploy patches to selected workstations, which run the PatchLink agent software. Using this client/server architecture, the IT administrator can very easily see which computers have (or lack) which OS or application patches, and can deploy those patches to the computers in the time and the manner that we choose.
Since it employs Web-based services, the solution is not restricted to the Local Area Network, or even the corporate WAN. The client computers can literally be anywhere on the Internet, and still access this patch management service.
Instead of trying to mimic other patch-deployment systems, Novell decided to choose a best-of-breed solution, and license the PatchLink Update Server, re-branding it as Zen Patch Management (ZPM). While the solution and the underlying software in both cases (PatchLink and Zen Patch Management) is exactly the same, for ZPM the licensing and the support (as well as the logo) come from Novell.
Microsoft and other application vendors find deficiencies or vulnerabilities in their software, and then develop software “patches” to fix the problem. These fixes undergo thorough internal testing (at least, we hope they do) by the application vendor, and then are released to the public. The engineers at PatchLink then test out the vendor-supplied patches, and “package” them for ease of deployment using the PatchLink Update Server, and then thoroughly test them again, before releasing them to their customers.
The PatchLink (and ZPM) Update Server is in constant contact with the PatchLink Internet servers, and are thus made aware of the availability of the new patches (over 2,000 currently available). The available pre-packaged patches can then be download (or “cached”) to the local PatchLink server by administrative control, and then tested (you do test your patch deployments, don’t you?) and deployed by the PatchLink administrator.
The ZEN Patch Management Web Interface
Zen Patch Management (or PatchLink) not only allows you to deploy these pre-packaged patches from Microsoft, Novell, HP, Symantec, Adobe and other vendors, but you can also create and deploy your own packages as well, using the Package Editor wizard (see “Roll Your Own” https://www.novell.com/coolsolutions/appnote/15409.html). This way, you can use ZPM or PatchLink to deploy your own, locally-created software deployment packages or updates, or even registry edits.
ZPM or PatchLink Update Server is installed on a Windows 2003 (SP1) server platform, and is quite easily installed, once you’ve mastered the skills of assigning of an IP address, keying in the registration code, and clicking the “Next” button. The workstation agent installation requires pretty much the same skills.
Patch Agent Deployment
But what if you need to deploy the PatchLink or ZPM workstation agent to 500 desktop computers? We can’t automatically deploy software packages using the PatchLink or ZPM Server, until we have these agents installed. So now what? Is it time to reconstitute the venerable “SWAT team”, and send out our small army of technicians to go around and install the ZPM agent to each individual computer?
The standard installation of the ZPM agent can be accomplished at the workstation by directing a Web browser to the ZPM server, click the “Devices” tab at the top of the Web page, and then click the “Install” button at the bottom of this page. This opens a window which allows you to download and run the Single Agent Installer, which is in the form of an .MSI file. To then perform the installation you have to accept the license agreement, enter the URL or IP address for the ZPM or PatchLink server, and the registration serial number for that server, before you can click “Install”.
This means manual intervention for each computer to be managed. YIKES! Isn’t there a better way? Fortunately, the answer is “YES!” – Cool Solutions is all about finding that better way.
To perform a silent, preconfigured and automatic installation of the PatchLink or ZPM agent, we just need to script it with a few parameters. This eliminates the need for entering the server URL and registration code, and such. We can write a small script to pre-supply the correct parameters, such as:
msiexec /i "C:\UpdateAgent.msi" /qn HOST="http://myServer" SERIAL="88888888-88888888" USEPROXY=0 GROUPLIST="GroupName1;GroupName2;GroupNameN"
If your users have to go through a proxy server to get to the PatchLink or ZPM server, you may have to script something like this:
msiexec /i "C:\UpdateAgent.msi" /qn HOST="http://myServer" SERIAL="88888888-88888888" USEPROXY=1 PROXYURL="http://myProxy" PROXYUSER="ProxyUser" PROXYPASS="ProxyPassword" GROUPLIST="GroupName1;GroupName2;GroupNameN"
Obviously, just replace the “http://myServer” with the actual URL for your ZPM or PatchLink server, replace the bogus serial number with your real registration code, and replace the GroupNames with groups that you have created on the ZPM or PatchLink server. The groups allow you to segregate and group computers not only by operating systems (default), but also by site, or department, or by VLAN, or by horoscope sign – whatever you choose.
NOTE: For more on the Zen Patch Management Agent installation, see https://www.novell.com/documentation/zenworks7/pdfdoc/ptm7install/ptm7ainstall.pdf
This solves the need for manual installation of the agent. But now you still have to get the .MSI installation file and the script to the local computer, and also have a means to execute the script. What to do now? Well, if your users are logging into a NetWare server, we can easily use the NetWare login script (the same applies if they are logging onto a Microsoft server, but we won’t go there now).
It is an easy matter to place the .MSI file and the script file in the PUBLIC folder on the server, let’s say in SYS:\PUBLIC\PATCH (or anywhere else that is publicly available to your users). Then we would want to put just a few conditional statements in the script to control the installation a bit more, such as:
REM SilentInstall.cmd @ECHO OFF IF EXIST "C:\Program Files\PatchLink\Update Agent\DAgent.exe" GOTO NOGO :GO msiexec /i "\\SERVER\SYS\PUBLIC\PATCH\UpdateAgent.msi" /qn HOST="http://patchserver.test.org" SERIAL="12345678-ABCDEFG" USEPROXY=0 GROUPLIST="ITDEPT;COOLGUYS" ECHO Zen Patch Management Agent installation completed on %date% %time% > c:\zenpatch.txt GOTO FINISH :NOGO ECHO Zen Patch Management Agent is already installed. >> c:\zenpatch.txt :FINISH
LAMER NOTE: The “.cmd” extension of this text file identifies it to NT/2000/XP as a command file. It could just as easily have a “.BAT” extension, and be called a batch file.
This little script just checks to see if the agent is already installed, and if not, installs it silently, pre-configured for the server and automatically inserting the workstation into the management groups that we specified. Then it writes a little text file to specify that it has performed the installation.
Of course, always test out all of your scripts on a test machine, before trying them out on real, live, breathing users (We don’t want them to become real, live, cursing users, now do we?). It’s definitely a drag to see your script fail because of a missing semicolon; it’s doubly a drag when other people see it fail because we didn’t test it first.
Using the Login Script
So once we have a working script, we want automatic execution of the script – this is where we want to use the NetWare (or other) login script. For better control, it is wise to create an NDS group for the users to which we wish to deploy the ZPM agent, and then put another conditional statement in the script to control the deployment. Such as:
IF MEMBER OF "PATCHAGENT" #\\SERVER\SYS\PUBLIC\PATCH\SilentInstall.cmd END
LAMER NOTE: The “#” character at the beginning of the line tells the NetWare client that this is an external command.
When all goes according to plan, the script installs the agent quite silently, and without rebooting, and the agent even reports in to the PatchLink or ZPM server, and is included in the specified groups (make sure the appropriate groups are created at the server beforehand). Before long, the agent will perform the basic “Discover Applicable Updates” (DAU) routine, and then we should know what operating system patches this workstation does – and doesn’t – have.
OK, that’s sweet – but wait a minute – aren’t we assuming that we can run this script on the local workstation? OOPS – that’s right. This will work just fine when the local user who logs onto XP has Administrative rights to the workstation. But our workstation best practices tell us that the end user shouldn’t have such rights, and should log on as a “limited” user, without the permissions to install new software.
In small environments, the end user may often be given (default) Administrator rights on the workstation. That means that when the user downloads and installs “Goofy Santa Screensaver” (insert your “favorite” malware here), and then the system becomes so bogged down with viruses and spyware that it becomes unusable, then we’ll just have to deal with the aftermath.
But in larger (or more secure) environments, we definitely want to limit the damage that an end user can inflict upon his system, so normally the end user is only able to log onto the local workstation with limited permissions – which exclude the installation of software. Now they have to call upon the PC Administrator to install any new software.
This security measure has its benefits, to be sure, but it also hampers our goals of automatically deploying the PatchLink or ZPM agent to the workstations. Once the PatchLink agent is installed, it runs as a local service on the workstation, so that the future software patches can be automatically installed without this issue of needing to use an administrative account. So the clincher is: how to get that agent installed without logging onto each PC as the Administrator?
Fortunately, Microsoft recognized this peril, and created the RUNAS utility, which allows us to run a program, using different credentials than the one used to logon to the desktop (i.e., the end user’s). This allows us to run a program, but run it “as” a different user, and then supply that (administrative) user’s credentials – username and password. Unfortunately, however, the RUNAS utility doesn’t work so well when the executable is remote to the local PC – as in residing on a volume on the server. This frustrates our mission to push out the ZPM agent to the workstations automatically.
Some admins have overcome this difficulty by first copying the installation file (UpdateAgent.msi) and script (SilentInstall.cmd) to the local PC’s disk first (perhaps to C:\TEMP), then running the script using the RUNAS utility to install the agent, and then running another routine at the end to clean up the locally-copied installation files. This can work, but it seems to me to create some unnecessary effort, and use unnecessary disk space, as well as creating unnecessary network traffic.
Another way to accomplish this is by using a third-party utility, such as CPAU.EXE. This is a freeware utility that was designed as an alternative to RUNAS, and will work nicely for our present needs (see http://www.novell.com/coolsolutions/tools/14460.html). We would copy CPAU.EXE to the same folder (e.g. SYS:\PUBLIC\PATCH) where we have the UpdateAgent.msi and script file. To use CPAU we would then have to modify the login script as such:
IF MEMBER OF "PATCHAGENT" #\\SERVER\SYS\PUBLIC\PATCH\CPAU -u %computername%\Administrator -p password -ex \\SERVER\SYS\PUBLIC\PATCH\SilentInstall.cmd END
NOTE: If the computer is a member of a domain, the “-u” switch can be followed by domain\username, to use the domain account instead of a local account.
This login script command now uses CPAU to pass the local PC Administrator’s credentials to execute the SilentInstall.cmd script, which installs the PatchLink or ZPM agent. The example above assumes that the local Administrator is named “Administrator”, and the password is “password” (obviously, this example is not terribly secure).
However, with this solution, the local PC admin’s name and password are stored in clear text in the login script – and a clever user might find out how to access and read the login script, thus potentially gaining admin control over the PC.
To overcome this vulnerability, CPAU has the capability to encrypt its commands into a job file, thus hiding the actual commands from prying eyes. To prepare an encrypted job file, we first use the CPAU utility (just once) with these parameters:
CPAU -u %computername%\Administrator -p password -ex \\SERVER\SYS\PUBLIC\PATCH\SilentInstall.cmd -file zpminstall.job -enc
The extra switches “-file
Not very useful for any prying eyes. And then to execute the encrypted job file, now we would change that command in the NetWare login script as such:
IF MEMBER OF "PATCHAGENT" #\\SERVER\SYS\PUBLIC\PATCH\CPAU -file \\SERVER\SYS\PUBLIC\PATCH\zpminstall.job -dec END
This tells CPAU to decrypt and execute the coded file as above, which includes the PC Administrator’s username and password to run “SilentInstall.cmd”. It accomplishes exactly the same task as the previous version above (execute SilentInstall.cmd with the local Administrator’s credentials), but it does so without potentially revealing what those credentials are.
Once the PatchLink or ZPM agent is installed on the workstation, then the fun really begins. The capabilities of the PatchLink or ZPM Server to determine which computers need which patches, to silently and persistently install those patches for you, and give you robust reporting on the patch status of all your computers – well, it’s pretty amazing.
For example, while editing this article, I went to the Web interface on my test PatchLink server, and saw that there were 37 Critical Microsoft Windows XP (post-SP2) patches needing to be applied to a test workstation with the PatchLink agent. I selected all of the patches, and clicked “Deploy” to start the Deployment Wizard. I accepted all of the defaults (“Next, Next, Next”), initiated the patch deployment, got a cup of coffee, and checked my email.
Whenever possible, the PatchLink agent will “chain” the patch deployments which normally require a reboot of the system, allowing you to deploy several “reboot” patches in a row, and then finally reboot at the end. So before I finished my coffee, the PatchLink agent on the test workstation notified me that it was time to reboot, so I clicked the “Reboot” button on the notification. Once restarted, the workstation came up with all of the operating system patches applied. What could be easier than that?
Just to recap the major steps we performed to automatically push out the PatchLink (or ZPM) agent, we did:
- Download the UpdateAgent.msi file to a user-accessible volume on your server
- Create a script file (e.g., SilentInstall.cmd) to control the installation of UpdateAgent.msi
- Download the CPAU utility to run the install with local Administrator permissions
- Create an encrypted job file using CPAU to hide the Administrator credentials
- Create a user group and a login script entry that will execute CPAU with the encrypted job file
FINAL NOTE: Before you do any of this in a production environment, on production users, make sure you have done it (repeatedly) in a test environment, with test users. Remember, we like to keep all of our “OOPSes” to ourselves, don’t we?
The automation of operating system and application patching is designed to take a lot of the repetitive drudgery out of our jobs, as well as to simply make sure that the job gets done. Sometimes, just getting the environment set up for the automation of software patching can be a bit onerous, especially if it means even a one-time visit to touch every workstation. My goal for this article is to hopefully reduce that drudgery, and free up some of your time for the projects and tasks that you really want to get done.