Novell Home

 

Before we get started with this “10 Tasks” list, I need to address the changes in store for Novell Connection magazine. Eric Schetselaar [I’ll leave it to the reader to decide how to pronounce his name. I tried once, but got in trouble.], the editor of this magazine, and I have talked at length about the new direction of the magazine and I’m chomping at the bit! Developing the content in both print and electronic media distracts from the content development time. And let’s face it folks—who has time to waste these days? The days aren’t getting any longer, but our To-Do lists seem to constantly be growing.

“If the only tool you have is a hammer, you tend to see every problem as a nail.”

—Abraham Maslow

Going to an electronic-only format allows the Novell Connection magazine team to devote their time and energy developing more video-based and audio-based content—from podcasts introducing product elements to videos that actually show you how to install or troubleshoot various products! Of course, we’ll continue the animated articles in the coming year and I heard something about a live polling system on the new site. (hmmm.... I hear one of the polling questions deals with my life ambition. Do you know the answer?) So best wishes to the Novell Connection magazine team during the conversion; I’m ready to join the project! Now let’s get to the meat of this article.

Every day IT professionals are bombarded with tasks that they might not have had time to master. In this article we focus on ten tasks that every troubleshooter should conquer primarily around the world of traffic analysis, which is, of course, a must-know field in general.

  1. Build a Baseline Library
  2. Build a collection of tools that run on a variety of platforms
  3. Write a concise report that management will actually read
  4. Perform trace back/reconnaissance on a target
  5. Relate your budget needs to the corporate bottom line
  6. Negotiate demo products from vendors
  7. Record your own screen videos
  8. Write for a publication
  9. Identify normal/abnormal TCP/IP communications
  10. Nurture a key law-enforcement contact

Let’s examine each of these tasks, their value to the IT troubleshooter and how you can become proficient in each of these areas.

> Task 1: Build a Baseline Library
In many cases it is difficult to tell what’s unusual in network traffic unless you know what is considered normal activity. For example, you might not know everything about a particular application you are running. But you do know that it relies on access to two servers during the start up process.

 

If the application loads slowly some day and your trace shows extremely high latency connecting to the second server, then you know where you ought to start looking to speed up the load process. A baseline library is a collection of trace files that depicts normal network or application operation.

It can save loads of time if I have a baseline to review when companies call me in to assist on troubleshooting their networks if the answer isn’t obvious or the faults appear to be at application level. Building a baseline library can be a very simple task and one that should be at the top of your list of things to do.

First, if possible, locate a “generic user system” (a system that is configured like most user machines on the network). If your sample user machine typically connects to the network via switch, hub out to the user system (or use a full-duplex tap if your link to the user system is full duplex). You could do some baseline captures directly from the user system, but you cannot do the user machine bootup and shutdown process baselines this way.

It can save loads of time if I have a baseline to review when companies call me in to assist on troubleshooting their networks if the answer isn’t obvious or the faults appear to be at application level. Building a baseline library can be a very simple task and one that should be at the top of your list of things to do.

To hub out to the user system, connect the user end of the network cable into a hub. Connect a second cable from the hub to the user system and a third cable to your analyzer. (See Figure 1.)

Note: You might want to apply a capture filter for all traffic to and from the user system so you don’t capture other broadcast/multicast traffic in your trace file. Although this would remove lots of extraneous traffic from your trace, you might miss some broadcast and multicast traffic that affects your user system such as some DHCP responses that may be directed to the broadcast address rather than your user system’s hardware address.

The Wireshark syntax for a capture filter based on hardware address is ether host 00:08:15:00:08:15.

What baselines should you capture? Here is a brief list of what I would consider “must have” baseline files:

  1. Bootup
  2. Login
  3. Logout
  4. Shutdown
  5. Application launch
  6. Application use (typical operations in separate trace files)
  7. Application shutdown
  8. Idle time
  9. Web browsing session
  10. E-mail send/receive
  11. Other key network functions

When you have gathered up your trace files, consider printing out the summary information for each; in Wireshark, select Statistics > Summary. As shown in Figure 2, the summary window provides you with general information on your captured traffic.

Another window you might want to print is the Protocol Hierarchy window (Statistics > Protocol Hierarchy). The Protocol Hierarchy statistics are important to review for your idle baselines or application baselines because they break down the various protocols and applications crossing the wire at this time.

When I create baselines, I use a binder with DVD insert pages. I print out the Summary statistics for each file, 3-hole punch the pages and store them immediately following the DVD page. As new applications are added, I insert new DVD pages and Summary printouts.

> Task 2: Build a Collection of Tools that Run on a Variety of Platforms
Each year I create and distribute the Laura’s Lab Kit (LLK) at various conferences; LLK first releases in March at Novell BrainShare. This DVD contains some of my favorite tools (albeit only for the Windows platform so far) and is a good starting point for building up your toolset.

Note: You can download the ISO image of Laura’s Lab Kit v8 from novell.com/connectionmagazine/laurachappell.html.

Your toolkit should contain enough variety that you can attack a problem from multiple angles. For example, if you believe the network problem is related to packet loss, you should have the analyzer to pick up the traffic and identify packet loss and retransmissions and you should have a tool that can stream data across that link to identify the frequency of packet loss and test your theory. Some of my favorite tools are listed below:

  • Wireshark
  • NetScanTools Pro
  • Ping Plotter
  • Sandboxie
  • Ettercap
  • Nmap
  • Nessus
  • Snort

Where can you get a comprehensive list of tools that should be in your toolbox? Fyodor (who created Nmap) has a fabulous listing of security tools at sectools.org. Each tool is described by one or more attributes. Check out Building The Mother Lode to see how they are listed on the sectools.org Web site.

For a list of tools that use WinPcap, the driver for link-layer access in a Windows environment, visit winpcap.org/misc/links.htm#tools.

Building a toolbox doesn’t have to be expensive; there are numerous open source tools that are available. Don’t overlook these tools just because they are free!

Recently, we had the joy of experiencing a small botnet attack that relayed spam through a rogue e-mail server we had running in our lab. By capturing the relayed e-mails and examining the source IP address and the e-mail header information, I could identify that the e-mails were being relayed through just two other e-mail servers.

> Task 3: Write a Concise Report that Management will Actually Read
In my early years of doing onsite visits, I would turn in a report to management that described every problem on the network in sequential order. The report might run 50 or 100 pages. After an onsite visit to a military installation, I was asked to come in for a debriefing session. The top brass wanted to know what the issues were in the most concise format possible: bullet point and prioritized. Not a single mucky muck looked at my beautifully printed and bound book that was thumped on the conference table in front of them. It was a doorstop to them.

Management typically doesn’t have the time to read the geeky details of their problem; if you have strep throat do you want to hear, “You have strep throat; take these antibiotics.” Or do you want to hear the details of the Group A streptococcus bacteria and its origin? Give me the darn meds, doc!

> Task 4: Perform Trace Back/Reconnaissance on a Target
Recently, we had the joy of experiencing a small botnet attack that relayed spam through a rogue e-mail server we had running in our lab. By capturing the relayed e-mails and examining the source IP address and the e-mail header information, I could identify that the e-mails were being relayed through just two other e-mail servers.

I turned off relaying on the lab server and performed some reconnaissance on the owners of the other e-mail servers to notify them that they also needed to turn off relaying.

Performing reconnaissance based on an IP address, e-mail header information, company/person’s name or domain name is a great skill to have.

NetScanTools Pro (netscantools.com) is the primary tool I use for reconnaissance and traceback. We have a demo version of this tool on the Laura’s Lab Kit v8. I suggest that the first time you run NetScanTools you check out the Help information. Yes, I know; we typically avoid Help information because the one thing it often doesn’t do is help. Kirk Thomas, the creator of NetScanTools Pro, did a great job here! In addition, check out the Wizard function; you can check off the information you know about the target and NetScanTools will list all the functions available to locate the source.

For more great information about reconnaissance and to see actual videos of me doing it, check out my many online animated articles for Novell Connection magazine. They even gave me my own landing page for the animated article series: novell.com/connectionmagazine/laurachappell.html.

 

> Task 5: Relate Your Budget Needs to the Corporate Bottom Line
If the boss doesn’t see money signs swarming around his head with the word “savings” intermingled, then you need to change your tactic when asking for money to pay for tools. Consider putting together a spreadsheet (complete with green color-coding; management may not understand the technology, but they know their colors!) that depicts the possible savings by resolving problems more efficiently.

You need to know some base information to build this spreadsheet. You need to know the average salary of the people affected by downtime. Bring that down to the cost/minute of downtime to realistically show the monetary effect of even the smallest instance of an unavailable network.

For every day on the job, you should have at least one hour of dedicated lab time.

Each time you troubleshoot the network, keep track of how much downtime was experienced and the time required to fix the problem. Now consider how much longer it would have taken to fix the problem without those tools. Document the imminent costs of an unresolved process and provide a one-page memo defining your cost savings for management.

> Task 6: Negotiate Demo Products from Vendors
You don’t have to pay for every product up front. Vendors who sell products that are truly worth their price want you to try their solutions. They are certain you will buy the product once you try it. The next time you consider purchasing some pricey solution, ask for a demo unit (hardware) or demo version (software).

This is where things can get really sticky: how do you get the time to work with all these great new tools? Here’s an issue you need to resolve with management: necessary lab time.

If your skills get old, then you won’t be as effective as you should be. How can you convince management that this is imperative? Begin documenting your research time whether it is looking for a new firewall configuration option, a new honeypot solution, application behavior or other task. Right now you do more research than management. And perhaps you realize that your learning is less effective when this research time is interspersed with your other duties. Consider revising your job description to include this research time to ensure it becomes part of your job definition.

> Task 7: Record Your Own Screen Videos
You are not just an IT troubleshooter; you are also a teacher. You need to educate your users how to perform various tasks on the network. You need to educate management on products and processes on the network.

You could write down your educational materials and deliver them in the traditional way: printed matter; however, consider how effective video-based training can be. All the Wireshark University self-paced materials and the Novell Connection magazine animated article series were created using Camtasia Studio from TechSmith Corporation (techsmith.com ).

Because I had no video editing experience, I really needed a tool that was easy to use and provided me with the ability to zoom in, apply call-outs (those pop-up messages) and produce videos in numerous formats. Figure 3.) shows Camtasia Studio during the creation of the September animated article on padding security issues.

Users and management will understand what you are teaching them when they can actually see you go through the steps first.

> Task 8: Write for a Publication
I was fortunate to get a chance to write for some of the early publications from Novell. Anyone remember the NetWare Technical Journal? (If so, you probably also remember GenOS, compsurf and NetGen!)

Writing for a publication opens the doors to numerous conferences that have press passes for writers and vendor information from manufacturers who want visibility for their products.

What? You don’t have anything to write about? Ha! Yes you do! Case studies are always fascinating to readers. They want to hear your situation, your thought process, your actions to resolve something, your results and finally everyone wants to know, in hindsight, what would you have done differently?

> Task 9: Identify Normal/Abnormal TCP/IP Communications
Now you probably know that I am a packet girl. I really think all IT troubleshooters should know how to identify normal and abnormal TCP/IP communications. Is something not working right in the protocol stack? If not, then you can’t blame TCP/IP. You need to look elsewhere.

Recently I received an interesting e-mail from Walt, a student who had attended my troubleshooting sessions at a conference. Walt said that seeing the packet-level analysis sessions “served to get me off dead center regarding my network management duties, set aside the $10,000 do-it-all packages and get back to basics. It’s a bit like the store clerk who never learned to add and subtract; they can’t make change without an electronic cash-register. We were in danger of falling into that pit regarding network troubleshooting.” I couldn’t have said it better, Walt!

Figure 4 depicts a problem related to TCP: although the user complains the network is slow, this trace provides evidence that the problem resides with the client’s system: the TCP receive buffer is full. The client’s system is saying “shut up” to the server. See! It’s not always the network’s fault!

Each time you troubleshoot the network, keep track of how much downtime was experienced and the time required to fix the problem. Now consider how much longer it would have taken to fix the problem without those tools. Document the imminent costs of an unresolved process and provide a one-page memo defining your cost savings for management.

This particular problem could only be identified by reading the packets on the wire. Consider how much time would have been spent running around to find the cause of poor performance if you didn’t just pop Wireshark on there to identify the issue!

> Task 10: Nurture a Key Law Enforcement Contact
At some point in your technology career you are probably going to need to talk with law enforcement. Who ya gonna call? Cold calling to a law enforcement division isn’t the best way to get the personal insight you should hope for. Now is the time to make those contacts in various law enforcement groups.

How do you do this? Well, many of you have heard me talk about the High Technology Crime Investigation Association (HTCIA). I’ve belonged to the group for years and have made tremendous contacts inside the FBI, Secret Service, District Attorney’s office and various cybercrime task forces around the world.

Now is the time to find these contacts. Remember: it’s not just a one-way street. What can you offer law enforcement? Can you be a resource on networking? Do you have a lab they can access? Can you make a presentation on some techie topic they might need?

You will not regret the time spent networking with law enforcement when you need some advice or assistance on a security issue.

So there you go—ten more items to put on your plate. Maybe this article is something to give to management; prioritize the topics you are most interested in and push for the time to put these tasks on the front burner, not in the storage room.

See you all on the electronic version of Novell Connectionmagazine! Make sure you’re subscribed to the new revamped e-zine at novell.com/connectionmagazine/subscribe.

Building the Mother Lode

So you’re looking to build the mother of all toolkits? Fyodor, the creator of Nmap, has done a lot of your work for you. On sectools. org, he offers a comprehensive list of security tools you should consider using to build your mother lode. Each tool is described by one or more attributes. Here’s how the attributes are listed on his site:

Did not appear on the 2003 list.
Popularity ranking rose / fell the given number since the 2003 survey.
Generally costs money. A free limited/demo/trial version may be available.
Works natively on Linux.
Works natively on OpenBSD, FreeBSD, Solaris and/or other UNIX variants.
Works natively on Apple Mac OS X.
Works natively on Microsoft Windows.
Features a command-line interface.
Offers a GUI (point and click) interface.
Source code available for inspection.

© 2014 Novell