If there is a runaway process called lookup-domain.p on my server, that typically this means there is something going wrong with Postfix.
First check if your server is not being spammed by checking how many messages are in the mail queue. Then see this article and start digging further.
https://www.virtualmin.com/node/22963
To get control back over your server, do:
- Reboot
- Stop Apache and MySQL
- Look hard with a magnifying glass