Internal Server Error

This document demonstrates debugging a web server returning HTTP 500 errors by investigating logs, configuration files, process information, and file permissions. Focus is on systematic investigation and root cause identification.

This document walks through debugging a web server returning HTTP 500 errors, demonstrating systematic investigation of logs, configuration files, running processes, and file permissions to identify and resolve the root cause.


Introduction

When a web server returns an HTTP 500 Internal Server Error, the root cause typically lies on the server side. This requires systematic investigation of logs, configurations, running processes, and system resources to identify the issue.


Initial Investigation

Reproducing the Error

The first step is to confirm the error by accessing the failing webpage. An HTTP 500 response indicates a server-side crash or misconfiguration, but provides no details about the underlying cause.

Checking System Logs

On Linux systems, logs are located in /var/log. Use date and time information to narrow the search:

 1# Check current date
 2date
 3
 4# Navigate to log directory
 5cd /var/log
 6
 7# List files sorted by modification time (most recent first)
 8ls -lt | head -n 10
 9
10# Check recent entries in syslog
11tail /var/log/syslog

If no relevant errors appear in system logs, the issue may be application-specific.


Identifying the Web Server

Using netstat to Find Listening Processes

When the web server software is unknown, identify which process is listening on port 80:

1# Find process listening on port 80
2sudo netstat -nlp | grep :80
FlagPurpose
-nPrint numerical addresses instead of resolving hostnames
-lShow only listening sockets
-pDisplay process ID and name for each socket

Example output reveals nginx is listening on port 80.


Configuration File Analysis

Locating Configuration Files

Configuration files on Linux are typically stored in /etc. For nginx:

1# Navigate to nginx configuration
2cd /etc/nginx
3
4# Check site-specific configurations
5ls /etc/nginx/sites-enabled/

Reviewing Site Configuration

Example nginx site configuration:

1uwsgi_pass 127.0.0.1:3031;

This indicates nginx is proxying requests to uWSGI (a common solution for connecting web servers to dynamic page generators).

Checking uWSGI Configuration

1# Navigate to uWSGI configuration
2cd /etc/uwsgi/apps-enabled
3
4# Review site configuration
5vi site.example.com.ini

Key configuration details:

ParameterValuePurpose
chdir/srv/site.example.comApplication directory
uid/gidwww-dataUser and group for running the app
wsgi-fileprod.pyPython script to execute
logto/var/log/site.logApplication log file

Enabling Debug Logging

Checking Application Logs

1# Check log file
2ls -lh /var/log/site.log

If the log file is empty or missing, enable debug mode in the application.

Modifying Application Code

Review the Python application:

1# Open application file
2sudo vi /srv/site.example.com/prod.py

Uncomment debug line:

1# Before:
2# bottle.debug(True)
3
4# After:
5bottle.debug(True)

Reloading the Service

1# Reload uWSGI to apply changes
2sudo service uwsgi reload

Root Cause Identification

Analyzing Debug Output

After enabling debug mode, reload the webpage. The error traceback reveals:

1PermissionError: [Errno 13] Permission denied: '/var/log/site.log'

Investigating File Permissions

1# Check log files
2ls -l /var/log/site*

Example output:

FileOwnerGroupPermissions
site.logrootroot-rw-r--r--
site.log.1www-datawww-data-rw-r--r--

The application runs as www-data but site.log is owned by root, preventing write access.


Applying the Fix

Changing File Ownership

1# Change owner to match application user
2sudo chown www-data:www-data /var/log/site.log

Verifying the Fix

Reload the webpage to confirm it now works correctly. The log file will populate with entries as requests are processed.


Long-Term Remediation

The immediate issue is resolved, but the root cause (why the file ownership was incorrect) requires further investigation:

  • Review log rotation configuration (/etc/logrotate.d/)
  • Ensure log rotation scripts preserve correct ownership
  • Add monitoring to detect permission issues before they cause outages

Debugging Workflow Summary

StepAction
1. ReproduceConfirm the error occurs
2. Check logsReview system and application logs
3. Identify processUse netstat to find listening services
4. Review configExamine web server and application configuration
5. Enable debugActivate verbose logging or debug mode
6. Analyze errorsRead tracebacks and error messages
7. Apply fixAddress the identified root cause
8. VerifyConfirm the issue is resolved
9. RemediatePrevent recurrence through configuration or monitoring

505 Internal server error

An internal server error is a Hypertext Transfer Protocol (HTTP) response, communicating to the user that an error occurred before completing a specific request. In short, the server tried to do something, but it failed. The most common internal server error is an HTTP 500 error, but other common error codes include 200, 301, 302, 304, 403, 404, and 503. Here is an example of what an internal server error might look like in a browser:

A Google website that displays a 500 error message informing the user that something is wrong with the server.

Why do internal server errors occur

Internal server errors can occur for different reasons, including various glitches in the website’s programming. Typically, this type of error occurs when there are bugs in the code. They can also occur if a database becomes unavailable, the server runs out of memory, or the application encounters a permission error. The downside to internal server errors is that they do not provide you with information on what exactly is wrong. As a developer, the error code only provides the general type of error occurring. Run different tests or set various parameters to try to reproduce the error. If you can reproduce the error, that will help you identify the issue and correct it.

Key takeaways

An internal server error is displayed when a web server cannot fulfill or complete a request. An internal server error informs you that an error exists but does not tell you exactly what the error is or where it is located. Troubleshoot by running tests to determine what and where the bugs are in your code and correct them to eliminate internal server errors.

Conclusion

Systematic investigation of web server errors involves checking logs, identifying running processes, reviewing configurations, enabling debug output, and analyzing permissions or resource constraints. Each step narrows the scope until the root cause is identified and resolved.


FAQ