The Linux logging system is a framework designed to collect, store, and manage logs generated by various system processes, services, applications, and kernel events. The Linux logging system helps system administrators monitor system activity, diagnose problems, and ensure security and performance.

Key Components of the Linux Logging System

1. Syslog: The syslog system is the core of Linux logging. It handles the collection and management of log messages from various services and applications. The syslog daemon (rsyslog, syslog-ng, or journald in modern Linux distributions) receives logs from the kernel, system services, and user applications.

2. Log Files: These are text files that contain messages about system activity. Most logs are stored in the /var/log/ directory.

3. Log Levels: Each log message in the system has an associated severity level, indicating the importance of the event. Common log levels are:

   • Emergency: A system is unusable.
   • Alert: Immediate action is needed.
   • Critical: Critical conditions like application or system failure.
   • Error: Runtime errors.
   • Warning: Warning conditions, less severe than errors.
   • Notice: Normal but significant conditions.
   • Info: Informational messages, no action needed.
   • Debug: Debugging messages, helpful for troubleshooting.

4. Journal: On newer Linux systems, especially those using systemd, the journald service manages system logs, replacing traditional syslog in some distributions. It stores logs in a binary format and provides tools to query logs efficiently.

Logging Process Flow in Linux

• Kernel and System Services: The kernel logs messages about hardware, device drivers, and low-level system events (e.g., dmesg).
• Applications and Daemons: Applications and system services (e.g., Apache, Nginx, SSH) generate logs that provide details about their operation.
• Log Collection: Logs are collected by syslog daemons or journald, which categorize and store them based on the severity level.
• Log Storage: Logs are typically stored in files located under /var/log/ or managed by journald in binary form.
• Log Rotation: Log files can grow large over time. The system uses log rotation tools like logrotate to manage and compress old logs, keeping disk usage in check.

Common Logging Daemons and Tools

1. rsyslog: A widely used syslog daemon in many Linux distributions.

   • Configuration files: /etc/rsyslog.conf, /etc/rsyslog.d/
   • Logs are written to /var/log/ by default.

2. syslog-ng: An alternative to rsyslog for centralized logging.

   • Configuration file: /etc/syslog-ng/syslog-ng.conf

3. systemd/journald: systemd's logging system stores logs in binary format.

   • Logs are stored in /var/log/journal/ (if persistent logging is enabled).
   • journalctl command is used to access logs.

4. Logrotate: A tool used to manage the size of log files by rotating them, compressing old logs, and deleting older logs.

   • Configuration file: /etc/logrotate.conf, /etc/logrotate.d/

Accessing Logs

1. Viewing Logs with journalctl (For systemd-based systems)

   • To view the most recent logs:

     journalctl -n 10
     

     This shows the last 10 log entries.

   • To follow log entries in real-time:

     journalctl -f
     

     This works like tail -f, showing new log entries as they are written.

   • To view logs for a specific service:

     journalctl -u <service_name>
     

     Example:

     journalctl -u apache2
     

2. Viewing Logs with cat, less, or tail

   • For logs stored in files like /var/log/syslog, /var/log/auth.log, or /var/log/messages, you can use:

     cat /var/log/syslog
     less /var/log/auth.log
     tail -f /var/log/messages
     

3. Log Rotation with logrotate

   logrotate ensures logs do not consume all available disk space. It is configured to rotate logs based on size, time, or custom criteria.

   • View logrotate configuration:

     cat /etc/logrotate.conf
     

   • Force log rotation:

     sudo logrotate /etc/logrotate.conf
     

Common Log Files and Their Contents

Here are some common log files in Linux and their typical content:

1. /var/log/syslog (General System Logs)

• Description: Contains general messages about the system, including startup, service activity, and system warnings.
• Example Output:

  cat /var/log/syslog

  Oct 6 09:22:16 servername systemd[1]: Starting Daily apt upgrade and clean activities...
  Oct 6 09:22:16 servername systemd[1]: Started Daily apt upgrade and clean activities.
  

2. /var/log/auth.log (Authentication Logs)

• Description: Tracks login attempts, sudo usage, and other authentication-related events.
• Example Output:

  cat /var/log/auth.log

  Oct 6 09:22:16 servername sshd[23567]: Accepted password for user from 192.168.1.1 port 22 ssh2
  Oct 6 09:22:20 servername sshd[23567]: Received disconnect from 192.168.1.1 port 22:11: Bye Bye
  

3. /var/log/messages (General System Messages)

• Description: Contains general system messages about the system, including kernel messages.
• Example Output:

  cat /var/log/messages

  Oct 6 09:22:16 servername kernel: [  188.177276] eth0: link up, 1000 Mbps full duplex
  Oct 6 09:22:18 servername systemd[1]: Starting Network Service...
  Oct 6 09:22:19 servername systemd[1]: Started Network Service.
  

4. /var/log/kern.log (Kernel Logs)

• Description: Contains logs related to kernel activity, including hardware events, driver loading, and kernel panics.
• Example Output:

  cat /var/log/kern.log

  Oct 6 09:22:16 servername kernel: [  187.233455] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x0)
  Oct 6 09:22:17 servername kernel: [  188.144601] ata1.01: failed to IDENTIFY (I/O error, err_mask=0x0)
  

5. /var/log/boot.log (Boot Logs)

• Description: Logs related to system boot, showing the processes and services started during boot.
• Example Output:

  cat /var/log/boot.log

  Starting LSB: Raise network interfaces...
  Starting OpenBSD Secure Shell server: sshd.
  

Advanced Logging Features

1. Log Filtering and Searching:

   • grep can be used to search logs for specific events or patterns.

     grep "error" /var/log/syslog
     

2. Centralized Logging:

   • For large infrastructures, you might use tools like Logstash, Fluentd, or Graylog to centralize logs from multiple systems.

3. Real-Time Monitoring:

   • Use tools like Logwatch or ELK Stack (Elasticsearch, Logstash, Kibana) to monitor and analyze logs in real-time.

Conclusion

The Linux logging system is an essential tool for system administrators, helping to monitor system health, debug problems, and ensure security. Log files are stored in /var/log/ and are categorized based on service or system function. With tools like syslog, journalctl, and logrotate, administrators can efficiently manage, view, and analyze logs to maintain system stability and security.