Network Performance Issues - The Top Ten List for Technicians

Print Version

7/20/05

AMS has analyzed numerous agency networks over the past 10 months and several trends have been noted. Some of these occur at almost all of the agencies and some are very exclusive at one or two of them. Listed below are some of the more dominate findings that could impact the performance of any LAN and/or WAN. Not all of these will apply to every agency or network. Some of the items listed below will still require a local technician with a good network monitoring tool and the knowledge on how to read the information gathered, to be on-site.

1. Ethernet Cables:

The Problem - The incorrect type or old Ethernet cables are critical in an agency that is trying to operate at a higher than 10baseT speed. In analyzing some agency networks, frequent retransmission errors are found to originate from specific workstations. The workstation(s) may also be experiencing errors in the application (i.e. AMS 360) due to this problem. In several cases, we have discovered Ethernet cables that are old, worn, bent or broken, bare wires, over long cable runs, or even incorrect cable types for the speeds the network is trying to run is the root cause for these performance problems. Mismatch Duplex settings (see below) may also manifest themselves as a bunch of retransmissions or even dropped connections.

Solution - Have the agency verify the all of the cable is compatible with the network, CAT5 or better for 10/100baseT or CAT5e or better for Gigabit networking. Replace any cables that are worn, heavy bends, run over frequently by chairs or other furniture or equipment, or those with bare wires.

2. Home / limited user class network devices:

The Problem - Network devices that were meant to be used in a home office type environment are showing up more often than we would like to see in our customer's networks. The attraction to this type of device is the price. They usually run less than $150 and appear to be just what a small agency needs to run their business across the Internet. Unfortunately, this is not the case. This type of device is made for occasional use. It’s unable to keep up with the high demands and constant use of a business, on a regular basis. This leads to slower handling of traffic, receive and transmit errors and even device failure. Some of this is transparent to the agency but most of the time it translates to sporadic poor performance at high demand times.

Solution - Examples of Home use brands to avoid include: Zonenet, Linksys, DLink, small Netgear devices, any device that cost them less than $150. The agency should look at devices that are comparable to the Cisco 800 series routers. This doesn’t mean they have to purchase Cisco brand but they will want to compare the model they are purchasing to this particular model, which would cost them more than $150 new.

3. Overworked Firewalls:

The Problem - When firewalls are required to scan each packet for email/virus and/or spyware/malware in addition to handling all Internet traffic for the agency, the device or appliance may have trouble keeping up and managing the network efficiently. This could easily overwhelm a firewall product or device causing delays in handling traffic in and out of the Internet. Firewalls that are asked to do too much for long periods of time or configured incorrectly can result in failures, service refusals and poor performance. Every process a firewall is asked to do, can be directly related back to the actual time it will have to handle each packet and could become a bottleneck in the network. Setting up specific firewall rules to help block/disallow certain types of traffic is more efficient than a generic policy. Your goal is to reduce the actual time the firewall needs to view, handle, or redirect each packet. The amount of Flash memory in a firewall appliance can make a difference as well. Cisco PIX firewalls for instance, come with limited amount of flash memory and when logging is turned on, this fills up quickly. It’s always in an agency’s best interest to err on the side of “more is better” when it comes to Flash memory on a firewall appliance, 8MB is not enough when logging events.

Firewall logging is a good way to know what’s going on but this will also take up resources. Some agencies have elected to write the logs on a server in the network, adding additional traffic to the LAN. The type of logging a firewall is doing can require more resources than intended, causing the device to work harder than expected and reducing the amount of resources available for traffic management.

Solution - The firewall fits in the same category as the small Home Use devices. Make sure the agency has the appropriate sized device for the amount of traffic it’s going to have to handle, looking at the recommendations for the amount of traffic. You will need a network monitor to determine whether the device is overloaded. Splitting the work up amongst other devices will also help prevent overload. Be careful using multiple devices as the firewall and router do need to work together.

4. Defective device ports / hardware

The Problem - When a device (router, switch, hub, firewall, server) is broadcasting incorrectly across the network, you can imagine the potential performance hit. Often having a bad port on a device can cause this type of network problem. Depending on the type of device, you may see information in the Event or Error logs on the device or the Domain Controller (if it exists), along the many communication errors or CRC errors within the network. Device resets, excessive collisions, failures, dropped packets or retransmissions all point to a problem. Isolating this problem is much more difficult as it can be masked by other noisy traffic contributors.

Solution – You could start with turning off particular ports or moving the network cables to a different port, if only a few of the workstations are having performance issues. If it’s the entire network, then borrowing a different device would be worth a try. Again, make sure the devices are the appropriate size and able to handle the amount of traffic the network is generating. Unmanaged devices are difficult to troubleshoot so it may take time to find the problem. Local network monitoring may be necessary to pin this issue down.

5. Active protocols that are not being used (IPX, LDAP, etc):

The Problem - Network traffic noise contributes to congestion and the quality of performance in a high volume network. Turning off all un-used, un-needed protocols will help limit the excess noise on a network and reduce that congestion. Printers, scanners and other devices are the major contributors to this kind of traffic and are often overlooked when it comes to optimizing a network infrastructure.

Solution - Turn off all un-used Protocols (such as IPX) on all devices and workstations, if possible. Make sure that there are no devices (as indicated) that still require the old protocol.

6. Bandwidth does affect overall performance:

The Problem - The agency replies that they have plenty of bandwidth and don’t understand why anyone would suggest they need more. In reality the download speed may be fine but upload speed isn’t adequate (1.5 mbps down vs 256 kbps up for example). Synchronous bandwidth is ideal, but if it is not available to the customer, then getting as close as possible is highly encouraged. T-1 type connections are synchronous and dedicated, where DSL and Cable connections can vary greatly. There is a difference between Business Class and Residential Class service as well for both DSL and Cable. Where Residential is cheaper, it’s also less reliable and almost always has a slower up than down speed. Communication providers won’t offer a Service Level Agreement for residential service so the agency may end up with a whole lot less than they paid for and it could be down more often then is reasonable.

Solution - AMS 360 utilizes as much bandwidth upload as it does download so it’s imperative that the agency is prepared for this with a solid Internet connection. This kind of problem can be very sporadic and make troubleshooting difficult.

7. Incompatible or old Network card drivers:

The Problem - Network cards will periodically have updated drivers that need to be applied to fix bugs or even add enhancements or features. Types of problems you will see with old/bad drivers will be similar to other problems. Intensive applications will generate network errors, disconnects, slow performance or slow downloads. Copy and paste would possible slow across the network.

Solution - As technology and software continues to change, so will the drivers. It’s a good idea to periodically check for updated drivers and apply them. Microsoft update will also supply new drivers for supported network cards. Remember, when a new driver is applied all settings will return to default and should be reset to what was configured previously.

8. Power management and Wake On-LAN enabled on Network Interface Card (NIC)

The Problem - Power Management of a network card is a great feature for a Notebook on battery power. For a workstation that is connected to normal electric power this is useless as well as a cause for performance issues. We have seen this cause some network cards to “fall asleep” while waiting for processes to complete and/or send back a return notice (during a long report for example). If it’s “fallen asleep” then a delay happens while waiting for it to wake up and the server session may time out rather than continue waiting. This translates into a performance hit right at the workstation (or worse, an application error). This applies to “WAKE ON LAN” settings as well. The network card awaits information, falls asleep waiting and then must be revived prior to it handling the return information correctly. Some Realtek network cards (very common) in particular have a problem waking up again in a timely manner. By the time they wake up, the packet was sent back as unable to be delivered and either the response is a Retransmission or stop error.

Solution - Turning off “WAKE ON LAN” and disable Power schemes would be highly suggested on all workstations and devices. To make the Power scheme changes open Control Panel > Power Options and change the power scheme to ALWAYS ON. To make a change to WAKE ON LAN; go to Network Connections, right click on Local Area Connections, select Properties. Click on Configure for the Network device, Advanced tab, Wake up Capabilities (wording may differ) change the value to OFF. There may also be a Power Management tab in this area. If yes, then select this tab and uncheck the "Allow this computer to turn off this device to save power". Ok to close all of the windows.

9. Network Interface Card (NIC) settings do not match the switch/hub in the network:

The Problem - A workstation or network with a mismatched duplex settings could cause a number of different types of errors. Some users will be unable to connect to the network at all. A user can ping an address but not browse. Others symptoms could be severe performance issues, disconnects, or errors in applications that are network intensive.

Solution - Speed and Duplex should always match the device in which they are connecting. For instance: An unmanaged switch is set to Auto negotiate. ALL Network Cards should also be set to AUTO negotiate. If the Switch is “Managed”, you have the option to hard set each port for any given device or workstation. To set the devices differently is to invite TCP Resets, Retransmissions and even lost or dropped packets. This will also decrease overall performance at the workstation in question.

10. Real Time Streaming Media

The Problem - Depending on the corporate structure and policies, Internet radio may or may not be allowed at the workstation. Any streaming product like Internet radio can greatly affect the amount of bandwidth available (network performance) but also the actual workstation performance.

Solution - If the workstation is having performance problems and the user is listening to Internet radio (or other streaming service), it would be advisable to turn Internet radio off while working in AMS 360 or other resource intense applications.

11. HTTP 1.1 settings in Internet Explorer (a bonus item to our top ten)

The Problem - Internet Explorer sets HTTP 1.1 as a default setting with version 4.0 and later which allows for a greater compression rate of AMS 360 data. If an agency has an older proxy server/appliance that’s not HTTP 1.1 compliant, this will respond with unreliable connections and/or random error messages. Another symptom of HTTP 1.0 is the compression rate of AMS 360 data is reduced by approximately 2% - 7%. This change will reflect in more bandwidth needed/used per transaction.

Solution - Have the agency make sure all proxy servers/appliances are compatible with HTTP 1.1 and the default setting in Internet Explorer at the workstations, has not been changed. They can check an option within Internet Explorer, at the workstation, to “Use HTTP 1.1 through proxy connections” found in the Internet Options, Advance tab that will force the proxy to keep this type of connection but if they are using an older proxy, it may refuse the connection. The agency should think about upgrading the proxy or removing it completely.