Palm Beach County Case Study
Novell Cool Solutions: Feature
By Christine McDermott
Digg This -
Posted: 27 May 2003
"This proactive monitoring has significantly reduced -- by greater than 80% -- calls to our help desk regarding DS related issues."
-- Craig Lessard, Novell Server Manager, Palm Beach County Board of County Commissioners
Palm Beach County's Board of County Commissioners' IS team has big plans for eDirectory.
After running the directory for more than nine years, serving more than 4,600 internal users, they've seen what it can do. From login and authentication to file and print and desktop management, the directory has served the Florida County well. Now they're ready to ratchet it up a notch. Putting eDirectory to the test will include connecting the countywide infrastructure using DirXML and leveraging eDirectory as a central Meta Directory for application query and identity management. The county's IS team will connect application and web servers, ERM databases and Oracle servers. They'll sync Active Directory with eDirectory. And they'll unite Palm Beach County's agencies, from the Tax Collector's office to the Clerk of the Courts, so that they can all communicate through one business-critical directory - the county's eDirectory Workforce Tree.
Few would argue that it's an aggressive plan. Even fewer would venture to try it. But, Palm Beach County is committed and confident. And its arsenal of directory management solutions from NetPro is one of the factors driving that confidence.
"I don't see how anyone could survive without them,' says Richard Schwartz, Server Manager for Microsoft Services on the county's IS team. "We can troubleshoot directory issues across the entire system, and in many cases, we can find the problems before our users do."
Palm Beach County has been using NetPro's directory management solutions for eDirectory since 2000 because they ensure 24x7 coverage of the county's central directory service and provide early warning of critical issues through comprehensive proactive alerting.
eDirectory Made Easy
Palm Beach County counts on NetPro's DS Expert and DS Analyzer to ensure the availability and performance of Novell eDirectory. The IS team also runs DirectoryAnalyzer, NetPro's award-winning Active Directory monitoring and alerting solution. According to Craig Lessard, Server Manager for Novell Services and a key member of the county's directory team, the Novell eDirectory solutions came first.
"We knew we were having directory issues, but we didn't know what they were or how to get to the heart of the problems," says Lessard. "We bought DS Expert and DS Analyzer as a package because they compliment each others' services. They work together and give us all the information we need."
DS Expert ensures the health of eDirectory by monitoring the directory 24x7 and proactively alerting on all key points of failure. DS Expert all but eliminates the need for time-consuming manual directory health checks and enables administrators to deliver consistent, reliable access to business-critical services.
DS Analyzer complements DS Expert by providing breakthrough directory management capability that enables administrators to analyze, troubleshoot and tune NDS eDirectory for optimal performance. DS Analyzer gathers directory traffic and utilization data from across the entire network and displays it in a graphical format from every possible perspective, including servers, partitions, containers, clients, objects, and NDS eDirectory background processes. With DS Analyzer, administrators can view directory traffic reports over any period of time and drill down on the data until problem areas are fully exposed.
The eDirectory team at the county believes so strongly in the two solutions together that it maintains a console that shows both solutions at half screen at all times. "It's the first thing I do every day," said Bruce Davis, Server Administrator at the county. "I check the console for alerts, and if I find one, I drill down on DS Analyzer to find the source of the problem. With DS Expert and DS Analyzer, we're getting the information ahead of our hardware monitors."
If it's Good For One Directory . . .
The Palm Beach County network environment includes Active Directory. It was introduced to provide a platform for web services and for the county agencies running pure Microsoft applications, including IIS, Apache and WebSphere. Today, Active Directory manages all Microsoft applications and Active Directory resources. And, like their Novell counterparts, the county's Active Directory team looks to NetPro to manage its directory.
Indeed, Schwartz says, the county's success with NetPro's Novell solutions drove the purchase of DirectoryAnalyzer for the Active Directory team. "The Novell team was really happy with the NetPro solutions," said Schwartz. "They said DS Expert and DS Analyzer had basically turned the network around, and we wanted to do the same for Active Directory."
Much like DS Expert for eDirectory, DirectoryAnalyzer ensures the reliability and performance of Active Directory by proactively monitoring and alerting Active Directory 24x7. DirectoryAnalyzer manages all of Active Directory's key infrastructure components, including replication, Active Directory-related DNS, domain controllers, operations masters, global catalogs, domains, and sites.
The goal, according to Schwartz, was to become more proactive in the management of the directory. Prior to DirectoryAnalyzer, Microsoft command line tool, REPLMON, was the only tool at the team's disposal. "We couldn't see squat," said Bill Cramer, Senior Server Administrator. "We could see there was an error, but we couldn't see what it was." Today Schwartz and his counterpart, Cramer, look to DirectoryAnalyzer's centralized view to see what's going on in Active Directory. "It's a huge help," said Cramer. "We can monitor and easily diagnose the health of the entire Active Directory environment from one central location. And we save all of the branch locations from worrying about it because we stay on top of their environments for them."
The very nature of Active Directory requires this highly proactive approach. "Active Directory could have errors and you wouldn't know about it for many months," explains Schwartz. "And then, when the problem hits, it can be really terrible." The county's Active Directory team uses DirectoryAnalyzer to anticipate problems and to run frequent tests to verify connections. "The troubleshooting tests are particularly valuable," he says.
Saving Real Dollars in Real Time
Real-time monitoring and troubleshooting make directory management far simpler for Palm Beach County's IS team. But, the true value of the solutions comes in the form of service to their internal customers. How? Dramatically reduced downtime, significant time and resource savings, and greatly enhanced productivity.
"We have a much more stable eDirectory network," says Lessard. "With DS Expert and DS Analyzer, we have less downtime and far better customer satisfaction. If we didn't have the software, we would have had outages that would cascade across the network. We wouldn't know how they happened and we wouldn't be in a position to catch them early."
According to Lessard, DS Expert runs continuous directory health checks, showing the directory's health 24 x7 and his team spends less time troubleshooting DS issues. Indeed, Lessard says, with DS Expert checking DS has become a daily task, rather than a monthly effort or a process that occurs only after problems are reported to the help desk. Without DS Expert, daily health checks would require manual processes and one-by-one access to many servers that host DS partitions.
"DS Expert makes continuous health checks possible," Lessard relates. "This proactive monitoring has significantly reduced -- by greater than 80% -- calls to our help desk regarding DS related issues."
Leading industry analyst firm, Gartner, places the cost of the average help desk call at $20 to $40 per call. So, with an 80% reduction in calls, the county is achieving a significant return on its investment in the solutions on that level alone. But, the savings do not end there.
According to Lessard, problem resolution time at the County has been cut by greater than 60%. "When we do get DS related calls our problem resolution is much more focused," he says. "DS Expert and DS Analyzer identify specific DS processes and the scope of a problem so that we can solve issues faster." With the average cost of a trouble ticket running upwards of $300 per ticket, a 60% reduction represents a significant savings for the county.
Employee productivity is another key area of savings, and it extends to the IS group and well beyond -- to the County's nearly 4,000+ network users.
"Although NetPro's tools have helped reduce the amount of time our technicians require for managing DS, the real impact of the software is reduced DS service outages and DS slowness," says Lessard. "An hour saved in DS problem resolution translates into hundreds of end users remaining productive with access to their networked resources. Our user community is the greatest benefactor."
Case in Point
In some instances, finding problems before they even impact the user demonstrates the greatest value of the products.
Palm Beach County's case in point occurred the very first week that Davis loaded DS Expert and DS Analyzer on the system. DS Expert pinpointed a replication slowdown with a red alert on the console, and the group put DS Analyzer to work on the cause. Acting much like a sniffer on the wire, DS Analyzer collected all of the incoming DS packets and enabled Davis to drill down on the problem. DS Analyzer probes the specific containers, servers, replicas, and clients creating the directory traffic, differentiating among 13 different types of DS traffic, including synchronization, bindery requests, backlinking, time, schema, etc. And, because the DS Analyzer agents had been installed on server1 early in the week, Davis had four days of data stored and could graph the partition.
"I immediately saw a distinct difference in the graph line for Server1," he said. "Server1's synch time was continually much higher than the other four servers holding copies of the replica. In fact, from 7 a.m. until 4 p.m. on Friday and Monday, there were in excess of 1 million requests every five minutes!"
To get to the real root of the issue, Davis took the 9 a.m. data point for Access, drilled down on the clients generating this traffic, and received a list of the top 10 clients generating the Access traffic. "The top client generating almost a million requests of Access traffic at 9 a.m. on Server1 was an IP address," said Davis.
The IP address didn't have a server name or a login name associated with it, but with a little research on the IP address, Davis was able to determine that the traffic was coming from an NT workstation that had a static IP address. Once he located the machine and physically pulled it off the wire, DS Analyzer immediately started graphing a downward line heading towards zero on that client. In addition, DS Expert showed Server1 completing its replica synchronization in two seconds. "To get to the bottom of the problem, it took me less than an hour," Davis said. "It was amazing!"
What's even more amazing is that he had no idea there was even a problem until he loaded DS Expert and DS Analyzer. "Prior to having this tool, we had no way to know the traffic spike existed, where slow DS response was coming from or how to fix it," says Davis. "It often took many hours to resolve slow DS response issues because we seldom knew what created it."
To the Future and Beyond
For nearly nine years, the Palm Beach County team of seven IS professionals has been ensuring the smooth operation of the county's directory and the services and applications that rely on it. They push out upgrades. Troubleshoot problems. Provide help-desk support for remote agencies. And they streamline the management of the network, actively leveraging the directory at its core.
If they get their way, the leverage they've achieved to date will seem like a walk in the park. By synching the entire network of systems and services to the eDirectory Work Force Tree using DirXML, they will drive the network user count well beyond 4,000 to more than 8,000 users. They will leverage the directory to personalize service for internal users, external users and citizens of the county. They will move dangerously close to the "nirvana" of single sign on. And they will get to the point where eDirectory will be a single point of failure. No worries on this team, though. With NetPro's solutions to manage their increasingly business-critical directory services, they say - bring it on!
Palm Beach County at a Glance . . .
Customer: Palm Beach County Board of County Commissioners
Number of Users: 4,600, climbing to 8,000 by project completion
NetPro Products Deployed: DS Expert, DS Analyzer, DirectoryAnalyzer
Benefits Achieved through Solutions:
- Dramatically reduced network downtime
- Improved network performance
- Enhanced end-user productivity and service
- Significant time and resource savings
- Shorter mean-time-to-repair
- Network efficiency improvements
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com