Hexamail Nexus Setup: Step-by-Step Installation and ConfigurationHexamail Nexus is an on-premises email security and management platform that provides filtering, anti-spam, archiving, policy enforcement, and secure web access for organizations. This guide walks through a complete installation and configuration process — from planning and prerequisites to advanced tuning and common troubleshooting. It’s aimed at system administrators deploying Nexus in a small-to-medium enterprise environment.
Before you begin: planning and prerequisites
- System requirements: Verify the target server meets Hexamail Nexus’s CPU, RAM, disk, and OS requirements. Nexus commonly runs on Windows Server (2016/2019/2022) or modern Linux distributions — check the vendor documentation for the exact supported versions.
- Network and DNS: Ensure you have static IPs for the Nexus server, proper DNS records for mail routing (MX, A), and relevant PTR records for reverse DNS if Nexus will send mail. Open required firewall ports: SMTP (25), SMTPS (465, if used), Submission (587), web UI ports (commonly ⁄443), and any additional ports for integration (LDAP, REST API).
- Mailbox and mail flow design: Decide whether Nexus will act as an inbound gateway, outbound gateway, or both. Plan MX priorities and whether mail will be routed through Nexus to your internal mail server (Exchange, Postfix, etc.).
- Authentication and directory: Prepare LDAP/Active Directory credentials for integration. Create a read-only service account if directory lookups are needed for policies or user-based controls.
- Certificates: Obtain TLS certificates for SMTP and the admin web UI. Use a trusted CA or your internal PKI; wildcard or SAN certs can simplify configuration if Nexus serves multiple hostnames.
Installation
- Obtain the installer
- Download the appropriate Hexamail Nexus package for your OS from the vendor portal. Ensure you have licensing details and admin credentials provided by Hexamail.
- Install on Windows
- Run the installer as Administrator.
- Follow prompts to accept the license agreement and choose the installation directory.
- Select components to install (mail gateway, web UI, indexing services, etc.). If unsure, install core components and add others later.
- At the end of the installation, choose whether to launch the Nexus service and open the web management console.
- Install on Linux
- Extract the package and run the provided install script (often something like ./install.sh) with root or sudo.
- Follow script prompts to set installation paths and service configuration.
- Configure the service to start on boot using systemd if the install script doesn’t do this automatically.
- Initial service startup
- Start or restart the Nexus service/daemon.
- Confirm services are listening on expected ports (use netstat/ss or Windows Resource Monitor).
- Check logs in the installation directory for errors.
Accessing the web management console
- Open a browser to the Nexus admin URL (e.g., https://nexus.example.local:443).
- Login with the default administrator account provided during installation or the one created by the installer.
- Immediately change the admin password and confirm TLS certificate warnings (replace default cert if present).
- Configure secure access: restrict the admin interface to trusted networks and consider enabling IP-based restrictions or two-factor authentication if Nexus supports it.
Basic mail flow configuration
- Define domains and hosts
- In the web console, add the domains you will manage (internal and any domains Nexus will accept mail for).
- Add internal mail servers as “next-hop” hosts for delivery (e.g., your Exchange or Postfix servers).
- Configure inbound SMTP
- Set Nexus to listen on the appropriate interfaces and ports.
- Configure banner and HELO/EHLO options to match your DNS/hostname.
- Enable and configure TLS for SMTP: bind the TLS certificate and select appropriate protocol and cipher settings.
- Configure outbound SMTP (relay)
- If Nexus sends mail outbound, define smart hosts or direct delivery settings.
- If authenticating to an upstream relay, configure credentials and ensure proper ports and TLS are used.
- MX and DNS changes
- Update public MX records if Nexus will be the primary inbound gateway.
- If Nexus is internal, update internal routing accordingly.
Directory integration and authentication
- Add LDAP/Active Directory connection in the Directory/Users section.
- Provide server address, base DN, service account credentials, and test the connection.
- Map directory attributes (email address, display name) so Nexus can resolve user identities for policies and quarantine notices.
- Configure SMTP authentication (if needed) using SASL mechanisms supported by Nexus and your clients.
Spam, virus, and content filtering
- Enable anti-spam engine
- Select the spam filtering engine (built-in, RBLs, or third-party).
- Configure spam thresholds and learning modes: start in a “tag-only” or quarantine-first mode to avoid false positives.
- Add public or private blocklists/allowlists for senders, IPs, and domains.
- Configure antivirus scanning
- Enable AV scanning and connect to any external AV engines if supported (ClamAV, commercial engines).
- Set actions for infected messages: quarantine, reject, or strip attachments.
- Content policies and DLP
- Create rules for attachment types (block executable attachments, large files).
- Configure data-loss prevention policies if Nexus supports content scanning — add patterns for credit card numbers, SSNs, or custom regex rules.
- Set quarantine and notification behavior for policy violations.
Quarantine, notifications, and user access
- Configure the quarantine store and retention policies.
- Customize quarantine notification templates and sender addresses.
- Enable user quarantine access (self-service) and integrate with directory so users can search and release messages.
- Set admin-level quarantine views and permission scopes.
Archiving and journaling (if applicable)
- If using Nexus for journaling, configure journaling rules to capture all inbound/outbound mail to the archive store.
- Set archiving retention, index options, and search capabilities.
- Ensure archived data is stored on reliable storage — use RAID, NAS, or SAN and consider off-site backups.
Logging, monitoring, and backups
- Configure log rotation and centralized logging (syslog or an external SIEM).
- Enable health checks and monitoring metrics; integrate with monitoring systems (Nagios, Zabbix, Prometheus) if Nexus exposes metrics or SNMP.
- Schedule regular backups of configuration and archives. Test restore procedures on a non-production system.
Security hardening
- Replace default certificates and change default ports if appropriate.
- Apply OS hardening best practices (disable unnecessary services, enable firewalls).
- Keep Nexus and the underlying OS updated with security patches.
- Limit administrative access by IP and use strong authentication controls.
Performance tuning
- Adjust thread pools, connection limits, and queue sizes according to message volume.
- Monitor CPU, memory, and disk I/O; increase resources or move services (e.g., indexing) to separate servers if needed.
- Tune spam/scan resources — for example dedicate CPU cores to AV scanning if supported.
Troubleshooting common issues
- Mail not accepted: check MX/DNS, listening ports, and firewall rules.
- Messages stuck in queue: inspect queue logs, check next-hop connectivity, DNS resolution for destinations.
- High false positives: loosen spam thresholds, enable learning, review blocklists and custom rules.
- AV false positives: update AV definitions or add safe-list rules for known-good senders.
- Web UI inaccessible: confirm service status, TLS certificate validity, and network ACLs.
Example: basic configuration checklist (quick reference)
- [ ] Server meets system requirements
- [ ] Static IP and DNS records (A/MX/PTR) configured
- [ ] TLS certificates installed for SMTP and admin UI
- [ ] Admin password changed and admin access restricted
- [ ] Domains and internal mail servers added
- [ ] SMTP inbound/outbound settings configured and tested
- [ ] Directory (LDAP/AD) integration tested
- [ ] Anti-spam and AV scanning enabled and tuned
- [ ] Quarantine and notification templates configured
- [ ] Backups, logging, and monitoring in place
Advanced topics and integrations
- High-availability: Configure failover and load-balanced Nexus instances for redundancy.
- Cloud connectors: Integrate with cloud services for backup, storage, or additional scanning.
- API automation: Use Nexus REST APIs (if available) for automation of rule changes, user management, or reporting.
- Migration: Plan cutover steps and coexistence when replacing an existing gateway — test mail flow with a subset of domains/users first.
Final notes
Deploying Hexamail Nexus successfully combines correct network and DNS design, careful policy tuning to balance protection with false-positive avoidance, and operational practices like monitoring and backups. Start conservatively with policies, validate with real traffic, and iterate configurations as you learn how Nexus interacts with your environment.