Table of Contents
Introduction
Writing a quick start article for an anti-spam gateway is not trivial. The fact of the matter is everyone’s idea on how to use email protection differs. Rather, this guide is my guide to sensible defaults in our environment. In our environment, we have the need to protect multiple email servers from spam and phishing. The direction is inbound. We generally trust our users outbound, but the internet is sending spam and we want to stop it.
Disclaimer: It’s impossible to stop spamming completely. There is no magic bullet. Phishing is even more difficult, because it’s perpetrated by criminals. As of 2024 criminals exploit the loophole of MAIL FROM: forgery and send basic text emails to avoid SpamAssassin score busting. So at the end of the day, if any mailbox is compromised, a phisher criminal will be able to attack using super basic techniques. Only rapid reporting and global coordination can succeed in stopping these bad actors. It’s cat and mouse all the time with criminals.
Steps
Install Proxmox Mail Gateway
As with all Proxmox systems, installation is trivial and I accepted all the defaults.
Note this important part of the installation manual though:
Recommended System Requirements
- Multi-core CPU: 64bit (Intel EMT64 or AMD64),
- for use in a virtual machine, activate Intel VT/AMD-V CPU flag
- 4 GiB RAM
In fact, I first tried with 2 GiB RAM, but got console warnings restarting certain services in the command line:
systemctl restart pmg-smtp-filter pmgpolicy postfix
After getting the memory warning, I immediately upped the RAM to 4 GiB. I also know from experience ClamAV demands at least 2 GB RAM to read it’s virus signatures and do comparisons in RAM, so 2 GiB simply won’t be enough.
I used an 8 GB disk. It’s wonderful to have such a small disk on a machine, but it makes perfect sense because it’s really just a store, check, and forward system in terms of space. I started off with RAID5 magnetic, but I noticed at MXToolbox SMTP delays so I migrated to NVMe.
Concepts
Internal SMTP versus External SMTP Port
The first concept that stumped me a bit is that there are two SMTP ports, ports 25 and 26 by default. On the forum this was clearly explained by Proxmox staff as an internal port, to receive email originating from your organization, and an external port, to receive email from outside your organisation.
The use of splitting these up is particularly useful for statistics and control – it’s good to know who is sending what and when, and if you love control, getting your own users to send via your gateway is incredibly useful. Also, if you have a big organization, you might want to know when one of your user’s mailboxes are compromised.
Do note though, for our application, we are only receiving from outside and sending to multiple insides. I tried for a short while to reconfigure Postfix domains on destination email servers to use the Proxmox Mail Gateway to also send email, but gave up after a little while because I didn’t want to break control panel configurations. I’ll post about this later when I have time to research and ask questions in the community and double check in a lab setup.
Enabling port 25 (for external)
One might ask why on earth, on sensible defaults, one would want to enable port 25 if email traverses email servers via port 25. The simple answer is security. There are broadly two types of security for new email servers, IP control and relay control. So, after struggling for a while, I noted that I had to include this:
Configuration=>Mail Proxy=>Networks
Wow dodgy! Does this mean you are creating an open relay? Thankfully not, because relaying is disabled by default.
Enabling Forwarding Domains
In our environment we have to on forward to multiple servers. For this to take effect, two additional settings were implemented, Relay Domains, and Transports.
Configuration=>Mail Proxy=>Relay Domains
Here I just add the domain, e.g. example.com
The second setting is this:
Configuration=>Mail Proxy=>Transports
Here I add the actual server name. In theory I could just use the domain, but to document the system better, and since multiple domains can reside on one server, I prefer the actual server name for example host.server.example.com
Voilà! At this point my anti-spam gateway starts doing its job.
Default Configuration after Installation
Since we’re expecting sensible defaults like all Proxmox products, the next part of this manual is about what was there already, and what did we decide to change. To keep my own sanity, here is a screengrab of an important configuration aspect and its installation defaults. We’re doing this screengrab because if we later mess around too much, as least we’ll know where we started. It’s really quite simple to over engineer spam management and end up with a broken system and enquiries about missing email.
As time went by though, we started asking hard questions about the defaults, and this is mostly documented in this article.
Checking the system’s logging out
The Proxmox UI has many places where you can track emails and view stats. These are all self explanatory and I’ll leave that for you to explore. For me though, there is nothing like tailing a log file. So this is how I started observing the email log file:
tail -f /var/log/mail.log
Yep! You’re reading this right. Proxmox Mail Gateway is based on the beautiful and resilient Postfix mail system and you can just tail the mail log file as you would normally. In fact, I’m glad I did this, because pretty soon on my new installation I saw this repeat:
RCVD_IN_ZEN_BLOCKED_OPENDNS
As an email specialist you probably already know what this means. In a nutshell you’re using 1.1.1.1 or 8.8.4.4 and you can’t use these open resolvers anymore for RBL checking, especially if you’re using the amazing Spamhaus.
At this point I was a bit confused though, because if you look at the above defaults, on my fresh system I didn’t have any DNSBL sites. So what gives? After much digging in the manual, I found two relevant sections.
- Don’t use 1.1.1.1 and an installation guide for
Unbound - Actually, this incredible configuration of Proxmox probably has a perfect
SpamAssassinimplementation, andSpamAssassinalready checksDNSBLs!
The important part of the Proxmox manual is here, and below that is an extract from the SpamAssassin manual:
https://pmg.proxmox.com/wiki/index.php/Getting_started_with_Proxmox_Mail_Gateway#Improving_Spam_Detection
Information on SpamAssassin:
Proxmox Mail Gateway offers two places where it can check information against DNSBLs:
…by the Spam Detector (SpamAssassin) – here the complete content of the mail (including potentially malicious URLs) is checked against a set of predefined lists
I’m pasting another screenshot from the manual, in case you haven’t read it. If you want to master this software, you have to master at least this section. The other reason why I’m pasting it here is because if you follow the SpamAssassin Apache DNS Block Lists Wiki link, you’ll see this:
- Spamhaus ZEN http://www.spamhaus.org/zen/ NOTE: Spamhaus is enabled as a “free for most” provider. See: http://www.spamhaus.org/organization/dnsblusage.html.
So that leaves the question. Are we using Spamhaus via SpamAssassin? Why does the manual still say we should use “quite good” DNSBLs e.g. Spamhaus?
With time, I’m sure we’ll start to understand which is which in this contradictory information.
Don’t use 1.1.1.1
First I tried my own DNS server which isn’t open. This didn’t work. But since they guide on installing XYZ is exceptional, pretty soon I had a caching DNS server on the same box. Now that I have (another) DNS caching server on my network it will be useful in the future to also open this to select servers. To be continued.
apt install unbound dnsutils
Check your installation:
ss -tulnp | grep :53 systemctl restart pmg-smtp-filter pmgpolicy postfix
Then Configuration -> Network/Time -> DNS – Add 127.0.0.1 as DNS
References:
- https://pmg.proxmox.com/wiki/index.php/DNS_server_on_Proxmox_Mail_Gateway
- https://forum.proxmox.com/threads/zen-spamhaus-et-uribl-too-many-queries-and-rcvd_in_dnswl_hi.125825/
Enable SSL for your end-users
The Proxmox Mail Gateway has a beautiful daily report that is sent to your end-users. This interface gives them the ability to quickly whitelist and blacklist specific addresses. Two, possible three, problems exist by default:
- If you haven’t set up SSL, they will get a certificate warning and be concerned.
- The FROM name and address might not be suitable for your environment.
- (Optional) If you don’t have a subscription, they will get alarmed with a No valid subscription dialog
Let’s deal with #1 first. We’re using Cloudflare for this to work, but other options also. See the embedded video further below.
Configuration -> Certificates -> Add an account “letsencrypt-live”
Then:
Configuration -> Certificates -> Add your domain and specify API as the usage method
Look at this great video if you want to see how to do this in totality:
In the next sections, we move on to the problematic Email ‘From:’ address, and then to the “No Subscription” issue.
Example of the Daily Reporting Email
The report looks something like this:
Changing the Postmaster@ address
Configuration -> Spam Detector -> Quarantine -> Email ‘From:’
No valid subscription?
Proxmox has open sourced their software and give an incredible value to end-users. We recommend you reward Proxmox back by buying at least a basic subscription. Go here for that:
https://www.proxmox.com/en/proxmox-mail-gateway/pricing
Post installation changes
Around a week after I did the initial installation I activated these additional settings:
Configuration -> Mail Proxy -> Options -> Use Greylisting forIPv6 from No to Yes
I did this because I see no point in having it active for IPv4 but not IPv6.
Configuration -> Spam Detector -> Options -> Use auto-whitelists from No to Yes
Apache SpamAssassin AWL seems well documented and very useful:
https://cwiki.apache.org/confluence/display/SPAMASSASSIN/AutoWhitelist
Just after I added this value, I found this from Proxmox staff, and put it back on No.
generally we recommend not using AWL and bayes filters, since they lead more often to problems than not. (we may even disable them by default on the next major release)
https://forum.proxmox.com/threads/awl-bayesian-filter-databases.126379/
Below auto-whitelists is Use Bayesian filter , for now I’ll wait till I have more info. Here is one definition:
https://cwiki.apache.org/confluence/display/JAMES2/Bayesian+Analysis
Issues
ClamAV
On my system, ClamAV Freshclam wasn’t started. I had to do this:
systemctl status clamav-freshclam
I also had to do quite a few EICAR tests before I could see ClamAV pick it up. It could simply be the way I tested, but I found one recent post about ClamAV on fresh installation and I suspect those users also had the same weirdness:
https://forum.proxmox.com/threads/emails-are-not-being-scanned-by-the-clamav-antivirus.134709/
The key to troubleshooting ClamAV with EICAR is quite simple by using any of these commands:
-
systemctl status clamav-daemonsystemctl status clamav-freshclamjournalctl -u clamav-daemonjournalctl -u clamav-freshclam
If things are working, you should see this:
2024-05-15T12:58:12.300544+02:00 anti-spam-gateway pmg-smtp-filter[7258]: 412CA6644954434F73: virus detected: Eicar-Signature (clamav)
Here is one EICAR link with which I didn’t have such good results:
How, just sending the usual email with the signature below did the job:
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
What are these perfect defaults?
That leaves us to guess what are the default protections? So far I count:
- Perfect installation of SpamAssassin including using their RBLs (?)
- Greylisting on IPv4
- Reverse check (but not strict hostname)
- ClamAV
- Daily summary reports to users about spamming
- Postfix, most defaults, including
- PTR must exist, however, not this setting:
- PTR is not set up A=>B, B=>A
- PTR must exist, however, not this setting:
Reject Unknown Clients?
Running a secure anti-spam email server means being very strict. On the default install, Reject Unknown Clients is off.
Finding concrete information on this is hard, because it’s a bit of a rabbit hole.
Reject Unknown Clients=rejectunknownsetting = actually Postfix’sreject_unknown_client_hostnamereject_unknown_client_hostnamein Postfix manual is a bit hard to follow:
https://www.postfix.org/postconf.5.html#reject_unknown_client_hostname
reject_unknown_client_hostname (with Postfix < 2.3: reject_unknown_client)Reject the request when 1) the client IP address->name mapping fails, or 2) the name->address mapping fails, or 3) the name->address mapping does not match the client IP address.
This is a stronger restriction than the reject_unknown_reverse_client_hostname feature, which triggers only under condition 1) above.
The unknown_client_reject_code parameter specifies the response code for rejected requests (default: 450). The reply is always 450 in case the address->name or name->address lookup failed due to a temporary problem.
For me the key was this paragraph: “This is a stronger restriction than the reject_unknown_reverse_client_hostname* feature”
What we know:
- Reverse is paramount in email security. So this default is always there.
- The
Reject Unknown Clientsis an even more rigid implementation, forcing email servers to adhere to perfect PTR security. In our opinion, this should also be a default so we changed it.
Now, in your log, you should see these events:
postfix/smtpd[12695]: NOQUEUE: reject: RCPT from unknown[172.xx.xx.xx]: 450 4.7.25 Client host rejected: cannot find your hostname,
In my opinion, this is probably one of the most important settings one should change!
*Reject the request when the client IP address has no address->name mapping.
Remaining Issues to Solve
MXToolbox Email Health
During routine testing, MXToolbox Email Health reported this on one of our receiving domains:
Warning – Does not support TLS.
To be honest, I don’t know where to start because apparently the default is to have this off:
As time goes by I’ll update this section. It sounds like TLS security should be enforced on incoming connections…
Postmaster address
Unfortunately the Proxmox Mail Gateway daily reports originate from a name and email address that we don’t like. I don’t know how to change that yet, in spite of going through the manual, all options in the UI, and looking on the forums.
Proxmox Mail Gateway <[email protected]>
Getting remote Postfix Servers to send via the Gateway
This didn’t work using either sender_dependent_default_transport_maps nor transport_maps
For example, I added data here but it didn’t work:
cat /etc/postfix/main.cf | grep transport sender_dependent_default_transport_maps = hash:/etc/postfix/dependent
smtpd_sender_restrictions
https://www.postfix.org/postconf.5.html#smtpd_sender_restrictions
Receiver Verification
We never enabled this, but it sounds useful:
Many of the junk messages reaching your network are emails to non-existent users. Proxmox Mail Gateway detects these emails on the SMTP level, before they are transferred to your network. This reduces the traffic to be analyzed for spam and viruses by up to 90% and reduces the working load on your mail servers and scanners.
Conclusion
This was by far one of the most fun and beneficial installations of any software I’ve done in many years. I love the quality you get when you use Proxmox. I often equate the Proxmox suite of software to focus and German engineering quality. It’s just so well written, so robust, so well backed up. Even their forum, it’s really such a professional high quality venue to get information and so well supported by their staff.