Kopano servers need memory optimization. There is an entire chapter written about it here:
https://documentation.kopano.io/kopanocore_administrator_manual/performance_tuning.html
Unfortunately Kopano open source is dead and they left their community in the lurch. Furthermore they supplement their attitude with additional statements like this:
There is also a more advanced tuning guide available to customers with a valid subscription. Please create a ticket with the Kopano support to retrieve it.
So you have to ask yourself why did they do this as open source in the first place? In the meantime, life goes on and we have to take care of our users whom we committed this product to.
With that out of the way, I’ll explain what we are going to do in this exercise. In this exercise, we’re going to performance tune a Kopano server for 75 users. We’ll use guidance from the article above and then feed that into Chat Omni and see if it can help us fast track the excessive amount of tuning and manual calculations that has to be done.
Afterwards we’ll check the validity of the AI and then repeat the exercise for 10 GB RAM.
Table of Contents
AI Prompt
Consider the memory requirements below, and given than I have 75 users and 6144 MB RAM, please help me with the exact parameters for performance optimisation of a Kopano server:
Kopano Guidance
9.2. Memory Usage setup
There are basically 4 [3] large parts of the server setup that use server memory:
- Kopano’s cell cache (caches individual cell data within a table view)
- MySQL’s buffer size (caches reads and writes from the ibdata file)
- MySQL’s query cache (caches exactly repeated SQL queries)
In a server purely running Kopano, make sure these caches are setup to use around 80% of the RAM in the server. The other 20% should be free for system processes, other processes (like MTA) and the webserver.
For a general rule-of-thumb, the following RAM distribution should be used:
Kopano caches:
- cache_cell_size: around 25% of total RAM size
- cache_object_size: about 100kb per user
- cache_indexedobject_size: about 512kb per user
These cache settings need to be configured in the /etc/kopano/server.cfg file. To activate the cache size changes the Kopano Server need to be restarted.
MySQL settings:
- innodb_buffer_pool_size: around 50% of total RAM size
- mysql_query_cache: 32Mb
- innodb_log_file_size: 25% of the innodb_buffer_pool_size
- innodb_log_buffer_size: 32M
- innodb_file_per_table: 1
- max_allowed_packet: 16M
- table_cache: 1000
These settings need to be configured in the /etc/my.cnf or /etc/mysql/my.cnf file below the [mysqld] section.
It’s recommended to change these MySQL settings before starting the Kopano Server and migrating user data.
AI Result
To optimize your Kopano server with 75 users and 6144 MB of RAM, we will allocate the memory according to the guidelines provided. Here’s a detailed breakdown:
Total RAM
- Total RAM: 6144 MB
- 80% of total RAM for Kopano and MySQL: 0.8 * 6144 = 4915.2 MB
- Remaining 20% for system processes: 6144 – 4915.2 = 1228.8 MB
Kopano Cache Settings
1. cache_cell_size
- 25% of total RAM size: 0.25 * 6144 = 1536 MB
2. cache_object_size
- 100 KB per user: 75 users * 100 KB = 7500 KB = 7.32 MB
3. cache_indexedobject_size
- 512 KB per user: 75 users * 512 KB = 38400 KB = 37.5 MB
Kopano Server Configuration (/etc/kopano/server.cfg):
cache_cell_size = 1536M cache_object_size = 8M cache_indexedobject_size = 38M
MySQL Settings
1. innodb_buffer_pool_size
- 50% of total RAM size: 0.5 * 6144 = 3072 MB
2. mysql_query_cache
- 32 MB (fixed)
3. innodb_log_file_size
- 25% of innodb_buffer_pool_size: 0.25 * 3072 = 768 MB
4. innodb_log_buffer_size
- 32 MB (fixed)
MySQL Configuration (/etc/my.cnf or /etc/mysql/my.cnf):
[mysqld] innodb_buffer_pool_size = 3072M query_cache_size = 32M innodb_log_file_size = 768M innodb_log_buffer_size = 32M innodb_file_per_table = 1 max_allowed_packet = 16M table_cache = 1000
AI Summary
The configuration values to be added to the respective configuration files are:
Kopano (/etc/kopano/server.cfg):
cache_cell_size = 1536M cache_object_size = 8M cache_indexedobject_size = 38M
MySQL (/etc/my.cnf or /etc/mysql/my.cnf):
[mysqld] innodb_buffer_pool_size = 3072M query_cache_size = 32M innodb_log_file_size = 768M innodb_log_buffer_size = 32M innodb_file_per_table = 1 max_allowed_packet = 16M table_cache = 1000
Analysis of the Actual Server
Let’s see if we can find those cache settings:
root@host:/etc/kopano# cat server.cfg | egrep "cache_cell_size|cache_object_size|cache_indexedobject_size" root@host:/etc/kopano#
Result: No settings found!
Let’s see if we can some of those MySQL settings (note, we’re using MariaDB):
root@host:/etc/mysql# egrep -r -i "innodb_|query_cache|max_allowed_packet|table_cache" /etc/mysql /etc/mysql/mariadb.conf.d/50-server.cnf:#max_allowed_packet = 16M /etc/mysql/mariadb.conf.d/50-server.cnf:#table_cache = 64 /etc/mysql/mariadb.conf.d/50-server.cnf:#query_cache_limit = 1M /etc/mysql/mariadb.conf.d/50-server.cnf:query_cache_size = 16M /etc/mysql/conf.d/mysqldump.cnf:max_allowed_packet = 16M
Result: This only looks slightly better, but as you can see some of the important ones are commented out.
Next, let’s log onto MySQL and at least see what the innodb_buffer_pool_size is as this seems pretty important. We’ll show you two ways, the “normal” way, and then a more verbose but human friendly way:
root@host:/etc/kopano# mysql -uroot -p ... Server version: 10.3.39-MariaDB-0ubuntu0.20.04.2 Ubuntu 20.04 ... MariaDB [(none)]> SHOW VARIABLES LIKE 'innodb_buffer_pool_size'; +-------------------------+-----------+ | Variable_name | Value | +-------------------------+-----------+ | innodb_buffer_pool_size | 134217728 | +-------------------------+-----------+ 1 row in set (0.002 sec) MariaDB [(none)]> SELECT -> VARIABLE_NAME, -> VARIABLE_VALUE, -> CONCAT(ROUND(VARIABLE_VALUE / (1024 * 1024), 2), ' MB') AS Value_in_MB -> FROM -> information_schema.GLOBAL_VARIABLES -> WHERE -> VARIABLE_NAME = 'innodb_buffer_pool_size'; +-------------------------+----------------+-------------+ | VARIABLE_NAME | VARIABLE_VALUE | Value_in_MB | +-------------------------+----------------+-------------+ | INNODB_BUFFER_POOL_SIZE | 134217728 | 128.00 MB | +-------------------------+----------------+-------------+ 1 row in set (0.001 sec)
Conclusion
As you can see, we have quite a lot of work to do. Tip to Kopano: For those paying clients, maybe a little script helper next time?
But before we go too far, let’s quickly see what MariaDB default is for innodb_buffer_pool_size.
# egrep -r -i "innodb.+default" /etc/mysql /etc/mysql/mariadb.conf.d/50-server.cnf:# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
Go figure! It’s spitting out 128 MB, but the help says 10MB?? What gives? All I can say for now, is welcome to the world of MySQL / MariaDB / Kopano, and performance tuning. Not 5 minutes of work.
Fixes
We first did innodb_buffer_pool_size and noticed a significant improvement using top and obversing the wa value.
root@host:/etc/kopano# egrep -r -i "innodb_buffer" /etc/mysql /etc/mysql/mariadb.conf.d/50-server.cnf:innodb_buffer_pool_size = 3072M
To be continued…
References
- https://documentation.kopano.io/kopanocore_administrator_manual/performance_tuning.html
- https://mariadb.com/kb/en/innodb-buffer-pool/