How to troubleshoot corrupted log rotation that deletes necessary logs or leaves oversized files on disk.
A practical, stepwise guide to diagnosing, repairing, and preventing corrupted log rotation that risks missing critical logs or filling disk space, with real-world strategies and safe recovery practices.
When log rotation behaves unexpectedly, system administrators face two troubling outcomes: essential logs vanish due to deletions or oversized files use up valuable disk space, complicating data retention and auditing. The root causes vary, from misconfigured rotation rules to broken post-rotate scripts, permission issues, or insufficient system resources at rotation time. To begin, inventory the current rotation configuration, verify the exact log paths being rotated, and confirm that the rotation tool version matches your deployment. A controlled test environment helps isolate issues without risking production logs. Documenting the baseline state will guide subsequent troubleshooting steps and prevent repeat problems.
Next, examine the rotation job itself for syntax or rule errors. In many systems, logrotate is driven by configuration files that specify retention counts, compression, and frequency. A misplaced semicolon, incorrect path, or conflicting include can cause rotations to skip or delete logs prematurely. Check for duplicate rules that collapse into a single action or override one another. Review file glob patterns to ensure they match only intended targets and do not accidentally include directories that should remain intact. After confirming the rules, simulate a rotation with a dry-run option if supported, observing which files would be affected.
Resource constraints and post-rotation scripts frequently drive corruption and data loss.
If simulated runs reveal unexpected deletions or oversized outputs, investigate the post-rotation commands. Scripts executed after rotation can accidentally purge logs or fail to relocate them correctly, leading to gaps or duplicates in the archive. Inspect permissions on the destination directories to ensure the rotating process can write and rename files without obstruction. Review any email alerts or logging hooks that indicate a failure, paying attention to exit codes and error messages. A failing post-rotate script can silently leave behind partial archives or trigger cascading failures that confuse monitoring systems. Correcting these scripts often resolves corruption without altering the rotation cadence.
Disk pressure during rotation is another common culprit. If the system temporarily runs out of inodes or writes buffers become full, the rotation might skip steps or truncate files unexpectedly. Monitor available disk space and inode counts in real time, especially on hosting machines with large numbers of compressed logs. Consider adjusting the rotation frequency or retention rules to reduce peak load, and enable backoff strategies so rotations don’t contend with unrelated heavy I/O operations. Implementing resource-aware limits helps keep rotation predictable, lowering the risk of corrupted outcomes during peak usage windows.
Validation and recovery practices reduce risk when corruption happens.
When you observe oversized logs after rotation, compression settings are a frequent suspect. The compression utility itself may fail to compress new archives, leaving uncompressed large files that degrade performance and complicate retrieval. Alternatively, a misconfigured compress directive could apply to files that should not be compressed, or produce corrupted archives due to interrupted processes. Audit the compression method, options, and the lifecycle of archived files. Ensure that the chosen algorithm matches your retention policy and that the decompression process remains viable for future audits. Establishing a predictable compression cadence reduces the chance of oversized, unmanageable logs.
Establishing reliable recovery paths is crucial when corruption occurs. Maintain off-site or secondary copies of critical logs to protect against accidental deletion. Implement checksums or cryptographic hashes for cohorts of rotated files to verify integrity during storage and retrieval. Keep an auditable trail of rotation events, including timestamps, file counts, and target locations. In the event of anomaly, a rollback plan should enable restoration from a known good backup state. This approach minimizes downtime and ensures that missing logs do not undermine compliance or incident response activities.
Change management and staged testing prevent cascading rotation failures.
When rotating rules are sound but behavior persists as faulty, consider environmental factors. Containerized environments, virtual machines, or chroot jails can introduce isolation boundaries that block expected access to log directories. SELinux or AppArmor policies may restrict file writes, causing silent failures. Review security policies to confirm that log directories are labeled correctly and that the rotation daemon has the necessary permissions across all stages of the process. Temporarily relaxing policies in a controlled test can help determine whether a security module is the root cause. If so, adjust the policy with precise, minimal changes to restore proper operation without broad exposure.
A well-documented change-management process supports stable rotations. Before deploying any fix, capture a precise change log describing what was altered, why it was changed, and how success will be measured. Communicate anticipated effects to stakeholders and schedule a maintenance window if possible to minimize the risk of cascading issues. Implement a reversible change where feasible, so you can revert quickly if unexpected side effects occur. After applying adjustments, re-run both dry and live rotations in a staged environment to validate that the logs are retained, not truncated, and that the disk usage remains within expected bounds.
Automation and monitoring collaborate to sustain healthy log rotations.
Beyond the mechanics of rotation, monitoring plays a pivotal role in early detection of anomalies. Deploy metrics that track the number of rotated files, the total size of archived logs, and the duration of rotation tasks. Set thresholds to alert on unusual changes, such as a sudden drop in rotated counts or a surge in file sizes that may indicate improper compression or retention. Centralized logging for rotation events helps correlate failures with system events, making it easier to diagnose root causes. Regularly review alert fidelity to avoid alert fatigue, ensuring that genuine corruption scenarios are surfaced promptly for investigation.
Automated diagnostics can speed up issue resolution by pinpointing failure points. Build lightweight health checks that run at predictable intervals, verifying that log paths exist, permissions are intact, and compression binaries perform as expected. When a problem is detected, collect a minimal, structured diagnostic bundle that includes configuration excerpts, sample rotated files, and recent rotation logs. This bundle can be shared with team members or vendors to accelerate debugging. By prioritizing automation, you reduce mean time to repair and improve overall reliability of the logging subsystem.
For legacy systems or heterogeneous environments, a phased approach to remediation helps maintain continuity. Start with a targeted fix on a single machine or a subset of log types, observe results over several rotation cycles, then roll out changes gradually. Avoid sweeping changes that affect all logs simultaneously, as this can obscure the actual culprit. Maintain multiple, independent backups in different formats and locations to tolerate diverse failure modes. A measured rollout minimizes production risk while delivering meaningful improvements in reliability and compliance across the fleet.
Finally, cultivate a culture of ongoing review and improvement around log rotation. Periodically revisit retention policies, compression strategies, and destination directories to ensure alignment with evolving compliance standards and storage capacities. Schedule routine audits that compare expected versus actual log inventories after rotations, and track any deviations to identify patterns. Encourage feedback from operators who experience the rotation firsthand, and incorporate their insights into subsequent iterations. By embracing proactive governance, you can sustain robust log rotation that preserves critical data and maintains disk health over the long term.
