Network Saturation

This document examines network performance through the critical concepts of latency and bandwidth, exploring how these factors determine data transmission speed across local and remote services. It covers diagnostic techniques for identifying slow network connections, strategies for optimizing performance based on data transfer patterns, and tools for managing bandwidth allocation through traffic shaping and connection monitoring.

Network Connections in IT

Working with Distributed Services

When working in IT, interaction occurs with services all over the Internet. At one moment, connection might happen to a service running on the local network, and the next might use another service running in a data center located on a different continent.

Service location diversity:

Perceiving Location Differences

If the network connection is good, it might not be possible to tell the difference where the website being browsed is hosted.

User perception of network quality:

Investigating Slow Connections

But if dealing with a network service that isn’t exactly up to speed, more details about the connection being used might be needed.

When to investigate network performance:

Understanding Latency and Bandwidth

Defining Latency

The two most important factors that determine the time it takes to get the data over the network are the latency and the bandwidth of the connection. The latency is the delay between sending a byte of data from one point and receiving it on the other.

Latency characteristics:

This value is directly affected by the physical distance between the two points and how many intermediate devices there are between them.

Factors affecting latency:

Typical latency by distance:

Defining Bandwidth

The bandwidth is how much data can be sent or received in a second. This is effectively the data capacity of the connection.

Bandwidth specifications:

Common bandwidth levels:

Bandwidth Marketing vs Reality

Internet connections are usually sold by the amount of bandwidth the customer will see. But it’s important to know that the usable bandwidth to transmit data to and from a network service will be determined by the available bandwidth at each endpoint and every hop between them.

Bandwidth bottleneck principle:

Real-world bandwidth limitations:

Latency and Bandwidth Interaction

Website Loading Example

To understand how latency and bandwidth interact, think about what happens when trying to visit a website over the Internet.

Website request lifecycle:

Cross-Ocean Scenario

If the web server is hosted somewhere across the ocean, the latency might be 100 milliseconds or so. That’s the time it takes for the request to reach the server.

Request-response with high latency:

The server will then generate a response and send it back. The first bytes of the response will again take 100 milliseconds to zap across the pond to the computer.

Bandwidth Impact on Download Time

Once the response is on its way, the time it takes for the rest of the data to arrive is determined by the bandwidth.

Bandwidth calculation:

If the available bandwidth between the two points is 10 megabits per second, 1.25 megabytes can be received every second.

Small Content Scenario

So for a website of about one megabyte of content, that large initial latency will be noticeable, since it’s an extra 20 percent on top of the total time to download it.

Small file download analysis (1 MB website, 10 Mbps bandwidth):

Large Content Scenario

But if the content is 10 megabytes or more, the initial latency will be less than five percent of the total time to download it, so it matters less.

Large file download analysis (10 MB website, 10 Mbps bandwidth):

Latency impact by file size:

Important

For small data transfers, latency dominates performance impact. For large transfers, bandwidth becomes the primary factor. Understanding which matters more for a specific use case is crucial for optimization decisions.

Optimizing for Different Scenarios

When to Optimize for Latency

Let’s say there’s an attempt to figure out why a network connection isn’t going as fast as desired. Remember that if transmitting a lot of small pieces of data, latency matters more than bandwidth.

Small data transfer characteristics:

In this case, ensuring the server is as close as possible to the users of the service is desired, aiming for a latency of less than 50 milliseconds if possible, and up to 100 milliseconds in the worst-case.

Latency optimization strategies:

Acceptable latency targets:

When to Optimize for Bandwidth

On the flip side, if transmitting large chunks of data, bandwidth matters more than latency.

Large data transfer characteristics:

In this case, having as much bandwidth available as possible is desired regardless of where the server is hosted.

Bandwidth optimization strategies:

Bandwidth Sharing and Competition

What Bandwidth Available Means

What is meant by bandwidth available? Computers can transmit data to and from many different points of the Internet at the same time, but all those separate connections share the same bandwidth.

Bandwidth sharing example (100 Mbps connection):

Uneven Bandwidth Distribution

Each connection will get a portion of the bandwidth, but the split isn’t necessarily even. If one connection is transmitting a lot of data, there may be no bandwidth left for the other connections.

Bandwidth competition scenarios:

Traffic Jams and Increased Latency

When these traffic jams happen, the latency can increase a lot because packets might get held back until there’s enough bandwidth to send them.

Congestion impact on latency:

Monitoring Network Usage

Personal Experience with Shared Bandwidth

This has probably already been experienced on personal computers. If several applications have been run using the same network at once, the overall connection speed may have seemed slower.

Application bandwidth consumption:

Using iftop for Monitoring

Processes using the network connection can be checked by running a program like iftop. This shows how much data each active connection is sending over the network.

iftop command usage:

 1# Install iftop (if not present)
 2sudo apt-get install iftop  # Debian/Ubuntu
 3sudo yum install iftop      # RHEL/CentOS
 4
 5# Run iftop (requires root)
 6sudo iftop
 7
 8# Monitor specific interface
 9sudo iftop -i eth0
10
11# Show port numbers
12sudo iftop -P
13
14# Show bandwidth in bytes instead of bits
15sudo iftop -B

iftop output interpretation:

1                  12.5Mb        25.0Mb        37.5Mb        50.0Mb
2┌─────────────────┴─────────────┴─────────────┴─────────────┴────────
3myserver.local    => cdn.example.com           15.2Mb  12.1Mb  10.5Mb
4                  <=                            1.5Mb   1.2Mb   1.1Mb
5myserver.local    => api.service.com            2.1Mb   3.5Mb   2.8Mb
6                  <=                            0.5Mb   0.8Mb   0.6Mb

Alternative network monitoring tools:

Multiple Users Impact

It might also have been noticed that the more users sharing the same network, the slower the data comes in. This is true for home connections and office connections alike.

Shared connection capacity:

No matter how much bandwidth is available, it’s a limited resource, so care will be needed with how it’s shared among users.

Note

Bandwidth is a finite shared resource. Even with high-capacity connections, simultaneous usage by many users or applications will divide the available capacity, potentially degrading performance for all parties when demand exceeds supply.

Traffic Shaping

Managing Bandwidth Distribution

If some applications are using so much bandwidth that others can’t transmit anymore data, it’s possible to restrict how much each connection takes by using traffic shaping.

Traffic shaping purpose:

Priority-Based Traffic Management

This is a way of marking the data packets sent over the network with different priorities.

Traffic priority levels:

Preventing Bandwidth Monopolization

To avoid having huge chunks of data use all the bandwidth, by prioritizing accordingly, processes that send and receive small packets can keep working fine, while processes that need the most bandwidth can use the rest.

Traffic shaping strategies:

Linux traffic shaping example (tc command):

 1# Limit outgoing bandwidth on eth0 to 1 Mbps
 2sudo tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms
 3
 4# Create priority-based queuing
 5sudo tc qdisc add dev eth0 root handle 1: prio bands 3
 6
 7# Assign SSH traffic to high priority (band 0)
 8sudo tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 \
 9  match ip dport 22 0xffff flowid 1:1
10
11# Show current tc configuration
12sudo tc -s qdisc show dev eth0

Traffic shaping benefits vs trade-offs:

Connection Limits

Maximum Connection Constraints

There’s also a limit to how many network connections can be established on a single computer. This isn’t usually a problem, but there could be bugs in the software that causes it to open way too many connections, or keep old connections open even if they’re no longer in use.

Connection limit factors:

Common connection limit issues:

Server Connection Problems

If this happens on a server, no new users will be able to connect to it until whatever is keeping those connections open closes them.

Server connection exhaustion:

Diagnosing connection issues:

 1# Count current connections
 2ss -s
 3
 4# Show all TCP connections
 5ss -tan | wc -l
 6
 7# Count connections by state
 8ss -tan | awk '{print $1}' | sort | uniq -c
 9
10# Show connections per IP
11netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
12
13# Check file descriptor usage
14lsof -u username | wc -l
15
16# Check system-wide file descriptor limit
17cat /proc/sys/fs/file-max
18
19# Check process-specific limits
20ulimit -n

Connection state monitoring:

Mitigation strategies:

Conclusion

Network performance in IT environments depends fundamentally on two factors: latency representing the delay between sending and receiving data affected by physical distance and intermediate hops, and bandwidth representing data transfer capacity that determines how much information can move per second, with internet connections sold by bandwidth but actual usable capacity limited by the slowest link in the entire path from source to destination. The interaction between latency and bandwidth determines overall performance differently based on data size—for small frequent transfers like API requests and chat messages latency dominates and servers should be placed close to users targeting under 50ms ideally and 100ms maximum, while for large transfers like video streaming and file downloads bandwidth becomes the primary factor requiring maximum available capacity regardless of server location, with the relative impact shifting as file sizes change from latency being 71% of total time for 100KB files down to only 0.25% for 100MB files. Bandwidth represents a limited shared resource where computers simultaneously connect to multiple services dividing available capacity unevenly, with high-volume connections potentially monopolizing bandwidth and causing traffic jams that dramatically increase latency from normal 20ms to over 500ms when fully saturated, observable through tools like iftop showing per-connection bandwidth consumption and alternative tools like nethogs for process-based monitoring. Traffic shaping manages bandwidth distribution by marking packets with different priorities to prevent huge data chunks from monopolizing capacity, allowing processes with small packets like interactive applications to maintain performance while bulk transfer processes use remaining bandwidth through strategies like rate limiting, priority queuing, and class-based management implemented via tools like Linux tc command. Connection limits on computers typically ranging from 1024 to 65535 file descriptors can be exhausted by software bugs causing excessive connection opening or failing to close old connections, manifesting on servers as inability for new users to connect until existing connections close, with diagnosis through commands like ss and netstat revealing connection states where thousands of ESTABLISHED connections indicate leaks and mitigation requiring connection pooling, proper cleanup code, timeout configuration, and monitoring alerts to prevent exhaustion scenarios.

FAQ