Background
On a Zabbix server, with seemingly defaults, disk space is continuously depleted. I guess it’s poor defaults or I installed it poorly. Either way, here are some tips to avoid you from hating this product as much as I do.
PS. It’s also got the most atrocious convoluted interface of any SNMP management product under the sun. Having worked with MRTG, PRTG, LibreNMS, and even RRDTool I can tell you categorically: It’s shit. Prepare to waste 100s of hours trying to find shit and configuring shit. Even more time and money hiring someone to do that for you. It might be open source and it might be “popular”, but my God, it’s terrible from a user-friendliness point of view.
History table grows proportionally large
NCDU is your friend here. Skip the drama and to NCDU at /var.
Now for some management tips to get rid of the nonsense history table. Be warned, it’s a complete nightmare. Once you’re fucked, you are completely fucked.
E.g. below, if things go fucked, don’t put `50000000` when deleting but put something like 50000 and then ramp it up from there…? What do I mean by fucked? Well, to delete 50 million rows from a history table in Zabbix takes fucking long and you’re going to be here for a while.
- 5000 rows = 2 seconds
- 50000 rows = 13 seconds
- 500000 rows = 2 minutes 30 seconds
- I guess you can see where we’re going
Also you’ll need a fuck load of disk space because the optimize command duplicate all the shit first. So instead of cleaning up the crap as below, just dump the shit.
Try clean up the crap
(Don’t do this on a slow disk, see next section)
SSH to server
cd /var ncdu mysql -uroot -p use zabbix; DELETE FROM history WHERE clock < UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 7 DAY)) LIMIT 50000000; optimize table history;
Next, go and find the fucking root cause:
HouseKeepingFrequency=1- ? Fuck that, it doesn’t work.
- `MaxHousekeeperDelete=5000`
- ? Not working
More hallucination bullshit that doesn’t work:
- `HistoryStoragePeriod=7d`
- `TrendStoragePeriod=90d`
- DBPartitionMode=partitioned
- ? Fuck that too.
- This directive doesn’t even exist
Dump the history tables
Stop wasting your time, just dump and recreate these 3 piece of shit tables:
WARNING: You’re about to trash your data for the sake of sanity.
Drop history
mysqldump zabbix history --where="clock > UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 1 DAY)) LIMIT 5000" > history_recent.sql mysql zabbix -e "DROP TABLE history;" mysql zabbix < history_recent.sql
Drop history_uint
mysqldump zabbix history_uint --where="clock > UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 1 DAY)) LIMIT 5000" > history_uint_recent.sql mysql zabbix -e "DROP TABLE history_uint;" mysql zabbix < history_uint_recent.sql
Drop history_text
mysqldump zabbix history_text --where="clock > UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 1 DAY)) LIMIT 5000" > history_text_recent.sql mysql zabbix -e "DROP TABLE history_text;" mysql zabbix < history_text_recent.sql
Zabbix MySQL tables fucked and you need the schema? Look here:
https://github.com/unix1986/zabbix/blob/master/database/mysql/schema.sql
Log File Rotation
Another fuckup of note, apparently log rotation needs a doctor’s degree to be activated.
Check the current log Linux logrotate rotation configuration here:
root@zabbix:/var/log# cat /etc/logrotate.d/zabbix-server-mysql
/var/log/zabbix/zabbix_server.log {
daily
rotate 14
size 100M
compress
delaycompress
missingok
notifempty
create 0664 zabbix zabbix
sharedscripts
postrotate
# make Zabbix reopen its logfile after rotation
systemctl reload zabbix-server >/dev/null 2>&1 || \
systemctl reload zabbix-server-mysql >/dev/null 2>&1 || true
endscript
}
Use this logrotate command to zero the log file:
logrotate -f /etc/logrotate.d/zabbix-server-mysql
Check if you have this directive:
# cat /etc/zabbix/zabbix_server.conf | egrep "LogFileSize" ### Option: LogFileSize # LogFileSize=1 LogFileSize=100