Improving your email reputation

In the past I have looked at, adding Sender Policy Framework (SPF) which looks at the bounce address of an email, I have also looked at SenderID which looks at the FROM address part of the email header, and then there was Domain Keys Identified Mail (DKIM) which signs your emails to confirm that they were indeed sent from an authorised email server (which has a published public key via DNS).

Now the final piece in the email puzzle is DMARC.

What is DMARC?

Very briefly it stands for Domain Message Authorisation Reporting and Conformance.

What does it do?

Two things;

  • Via DNS you (the sender) publish under your domain record an instruction that states whether or not a receiving email server should trust your SPF, SenderID and DKIM.  All three are also published under your domain DNS zone.  You can say whether or not they should reject or quarantine emails that purport to come from your email server but don’t (or do nothing).
  • Receiving information in XML format about how your domain reputation is fairing from a given receiving email server, as and when your users send emails to third parties.

What does the DNS record look like?

It looks like the following;

_dmarc IN TXT "v=DMARC1; p=none; rua=mailto:postmaster@lab.tobyheywood.com"

OK, so if we break this down a little we have the following components;

  • “_dmarc” – This is the name of the TXT record, which recipient email servers will try to retrieve from your DNS when configured to use DMARC.
  • “IN” – Standard part of a BIND DNS record.  Means Internet, nothing more, nothing less.
  • “TXT” – This is the record type.  DMARC utilises the bog standard TXT record type as this was seen as the quickest method to adoption, rather than treading the lengthy path towards a new DNS record type.
  • Now we have the actual payload (note they are separated by semicolons);
    • “v=DMARC1”  this is the version of DMARC
    • “p=none” – We are effectively saying do nothing and we don’t want to confirm or deny that the emails are from us or that the SPF, SenderID or DKIM information should be trusted
    • “rua=postmaster@lab.tobyheywood.com” – Doesn’t need to be included but if you do include this part, then you can expect to receive emails from the 3rd party email servers who have received email(s) from your domain, confirming what they thought of it

Are there other options?

Yes, though I am only going to focus on a couple here;

The “p=” option has three values that you can use;

  • none – Effectively do nothing, this should be set initially whilst you get things set up. Once you have confirmed things look good, then you can start to be a bit more forceful in what you would like other email providers to do with messages which do not come from your email server.
  • “quarantine” – This is were they would potentially pass the email on for further screening or simply decide to put it into the spam/junk folder.
  • “reject” – This is you saying that if a 3rd party receives an email, supposedly from you, but that wasn’t sent from one of the list of approved email servers (SPF or SenderID) or if it doesn’t pass the DKIM test then it should be rejected and not even delivered.

You set your _dmarc record, now what?

We will assume that you DNS zone has replicated to all of your DNS servers and that you have correctly configured the you email server to sign your outbound emails with your DKIM private key.

At this point I would highly advise going to https:///www.mail-tester.com and send a test email (with a realistic subject and a paragraph or two of readable text) to the email address they provide.

Once mail-tester.com has received your test email, they will process the email headers to confirm whether or not SPF, SenderID, DKIM and DMARC are all correctly configured and working.

It is possible that if your DNS servers are not completely aligned and up-to-date, mail-tester.com may be unable to provide an accurate report.  If that happens give it 12 hours and repeat the test again.

 

Credit: Thanks to Mr Darkroom for making the featured image called Checkpoint available on Flickr.

A step-by-Step Guide to Installing Spacewalk on CentOS 7

It would appear that during an upgrade of my blog at some point over the past year, I have managed to wipe out the original how to guide to installing Spacewalk on CentOS 7, so here we go again.

A step-by-step guide to installing Spacewalk on CentOS 7.  Just in case you weren’t aware Spacewalk is the upstream project for Red Hat Satellite Server.

Assumptions

  • You know the basic idea behind Spacewalk, if not see here
  • You have a vanilla VM with CentOS 7.2 installed which was deployed as a “minimal” installation
  • You have subsequently run an update to make sure you have the latest patches
  • You have root access or equivalent via sudo
  • You have got vim installed (if not run the following command should fix that)
    yum install vim -y
  • The machine you intend to install Spacewalk onto has access to the internet

Preparation

Firstly, we need to install and/or create the necessary YUM repo files that will be used to install Spacewalk directly from the Spacewalk official yum repository and all it’s associated dependencies.

  1. Run the following command as root on your spacewalk VM
    rpm -Uvh http://yum.spacewalkproject.org/2.5/RHEL/7/x86_64/spacewalk-repo-2.5-3.el7.noarch.rpm
  2. You then need to manually configure another yum repository for JPackage which is a dependency for Spacewalk, by running the following (you will need to be the root user to do this);
    sudo -i
    cat > /etc/yum.repos.d/jpackage-generic.repo << EOF
    [jpackage-generic]
    name=JPackage generic
    baseurl=ftp://ftp.rediris.es/mirror/jpackage/5.0/generic/free/
    enabled=1
    gpgcheck=1
    gpgkey=http://www.jpackage.org/jpackage.asc
    EOF
  3. And then we also need to install the EPEL yum repository configuration for CentOS 7;
    rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Installation: Embedded Database

Spacewalk utilises a database back end to store the required information about your environment.  The two options are PostgreSQL and Oracle.  Neither would be my preference but I always opt for the lesser of two evils – PostgreSQL.

The installation is a piece of cake, and can be performed by issuing the following command at the command line;

yum install spacewalk-setup-postgresql -y

During the process you should be prompted to accept the Spacewalk GPG key. You will need to enter “y” to accept!

Installation: Spacewalk

Now things have been made pretty easy for you so far.  And we wont stop now.  To install all of the required packages for spacewalk just run the following;

yum install spacewalk-postgresql

And let it download everything you need.  In all (at the time of writing) there were 379 packages totalling 563M.

Again you will likely be prompted to import the Fedora EPEL (7) GPG key.  This is necessary so just type “y” and give that Enter key a gentle tap.

And.. you will also be prompted to import the JPackage Project GPG key.  Same process as above – “y” followed by Enter.

During the installation you will see a lot of text scrolling up the screen.  This will be a mix of general package installation output from yum and some commands that the RPM package will initiate to set and define such things as SELinux contexts.

The key thing is you should see right at the end “Complete!”.  You know you are in a good place at this point.

Security: Setting up the firewall rules

CentOS 7 and (for that matter) Red Hat Enterprise Linux 7 ship with firewalld  as standard.  Now I’m not complete sure of firewalld but I’m sticking with it, but should you decide you want to use iptables (and you have taken steps to make sure it is enabled), then I have provided the firewall rules required for both;

firewalld

firewall-cmd --zone=public --add-service=http
firewall-cmd --zone=public --add-service=http --permanent
firewall-cmd --zone=public --add-service=https
firewall-cmd --zone=public --add-service=https --permanent

Note.  Make sure you have double dashes/hyphens if you copy and paste as I have seen the pasted text only using a single hyphen.

Skip to section after iptables if you have applied the above configuration!

iptables

Now as iptables can be configured in all manor or ways, I’m just going to provide the basics, if your set-up is typically more customised than the default, then you probably don’t need me telling you how to setup iptables.

I will just make one assumption though.  That the default INPUT policy is set to DROP and than you do not have any DROP, REJECT lines at the end of your INPUT chain.

iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

And don’t forget to save your firewall rules;

# service iptables save

Configuring Spacewalk

Right then, still with me?  Awesome, so lets continue with getting Spacewalk up and running.  At this point there is one fundamental thing you need…

You must have a resolvable Fully Qualified Domain Name (FQDN).  For my installation I have fudged it and added the FQDN to the host file, as I intend to build the rest of my new lab environment using Spacewalk.

So assuming you have followed everything above we can now simply run the following;

spacewalk-setup

Note.  The above assumes you have the embedded PostgreSQL database and not a remote DB, or the Oracle DB option.  Just saying.

So you should see something like the following (it may take quite some time for many of the tasks to be completed so bare with it);

[root@spacewalk ~]# spacewalk-setup
* Setting up SELinux..
** Database: Setting up database connection for PostgreSQL backend.
** Database: Installing the database:
** Database: This is a long process that is logged in:
** Database:   /var/log/rhn/install_db.log
*** Progress: ###
** Database: Installation complete.
** Database: Populating database.
*** Progress: ###########################
* Configuring tomcat.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
** GPG: Creating /root/.gnupg directory
You must enter an email address.
Admin Email Address? toby@lab.tobyhewood.com
* Performing initial configuration.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? 
** /etc/httpd/conf.d/ssl.conf has been backed up to ssl.conf-swsave
* Configuring jabberd.
* Creating SSL certificates.
CA certificate password? 
Re-enter CA certificate password? 
Organization? Toby Heywood
Organization Unit [spacewalk]? 
Email Address [toby@lab.tobyhewood.com]? 
City? London
State? London
Country code (Examples: "US", "JP", "IN", or type "?" to see a list)? GB
** SSL: Generating CA certificate.
** SSL: Deploying CA certificate.
** SSL: Generating server certificate.
** SSL: Storing SSL certificates.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? y
* Restarting services.
Installation complete.
Visit https://spacewalk to create the Spacewalk administrator account.

Now at this point you are almost ready to break open a beer and give yourself a pat on the back.  But lets finalise the installation first.

Creating your Organisation
(that’s Organization for the Americans)

Setting up your organisation requires only a few simple things to be provided.

  • Click the Create Organization button and you should finally see a similar screen to the following;
    Set up your Spacewalk organization.
  • The last thing to do now you have your shiny new installation of Spacewalk is to perform a few sanity checks;
    Successful installation of Spacewalk.
  • Navigate to Admin > Task Engine Status and confirm that everything looks health and that the Scheduling Service is showing as “ON”
  • You can also take a look at my earlier blog post – spacewalk sanity checking – about some steps I previously took to make sure everything was running.

And there we go, you have install Spacewalk.

Security Broken by Design

Admit it. You, just like me, use Google every day to answer those tough questions that we face daily.

Sometimes we will ask it how to get us home from somewhere we have never been before – “OK Google, take me home” – other times we might be close to starvation (relatively speaking) – “show me interesting recipes” or “OK Google, give me directions to the nearest drive through McDonalds”, but were I use it most, is at work, where I search for such mundane things as; “rsyslog remote server configuration”. Yes, I know, I could just look at the man page for rsyslog.conf but Google seems to have worked its way into my head so much that it is often the first place I look.

Right… back to the topic at hand – Security Broken by Design.

So whilst Googling how to set up a remote syslog server I read through one persons blog post and an alarm bell started to ring!

This particular post had correctly suggested the configuration for rsyslog on both the client and server but then went on (in a very generic way), instructing readers to opening up firewall ports on the clients.

This highlighted a fundamental lack of understanding on the part of the individual whose blog I was reading. You only need to open up ports 514/tcp or 514/udp to enable rsyslog to function on the server-side.  The connection is initiated from the client NOT the server.  Granted, in a completely hardened installation it is likely that outbound ports will need to be enabled.  BUT, where security is concerned, I feel that things should not be taken for granted or worse, assumed!

This generic discussion about security seems completely idiotic! The likes of Red Hat, Ubuntu and almost all other distributions now enable firewalls by default.  And the normal fashion for such a thing, is to allow “related” and “established” traffic to flow out of your network card to the LAN and potentially beyond.  But (and more importantly) to block none essential traffic inbound to your machine.

If you are working in a hardened environment then one of the two options below would be better suited for your server;

So in short.

Please think before you apply make potentially unnecessary changes to your workstations and servers!

Thanks to Sarah Joy for posting the featured image Leader Lock on Flickr.

DKIM + Courier-MTA on CentOS7

Email.  One of those things which is critical to business and depending on what you read and who you speak to, you may be led to believe that it is dying out in favour of instant messaging technologies.  Well, as of right now, I can’t see it dying out any time soon, though it does come with a really annoying, frustrating, excruciating problem.  Junk mail!  Otherwise know as spam!  It has once again become a bit of a problem for me, especially when the likes of yahoo begin to imped the flow of emails leaving one of the servers I have worked on recently unable to send emails to recipients with Yahoo accounts.

Now, don’t get my wrong.  I don’t hold a grudge with any of the large email service providers for blocking spam which has sadly managed to be sent via a mail server I have some involvement with, but I do wish the world wasn’t such a bad place that we have to continuously fight on our email frontiers to protect our virtual/real name and brand.

So anyway.  On with the show

The server

  • CentOS 7
  • Patched to the nines
  • Courier MTA
  • Blocking of known spammers was enabled
  • Spamassassin was used as a secondary measure should the hard blocks fail
  • SPF is configured for all hosted domains
  • DKIM was not configured
  • DMAC was not configured
  • It *was not* an open relay
  • The time was not in sync, which  didn’t help later down the line

The problem

The server’s “reputation” had been tarnished as someone had managed to start sending spam via the server through a vulnerability which was found in one of the websites hosted on the server.

After reviewing the advice from Yahoo, I started my investigation on how to approach this topic.

A bit of googling, led me to this web site; https://www.strangeworld.com/blog/archives/128. Now for me, there were a few things missing in the instructions, though complete were it was most needed.  So recreating some of the  good work done on the strangeworld.com website and providing my own twist on things here;

DKIM & ZDKIMFILTER Pre-requisites

In order to install zdkimfilter you need to manually compile the code and install (a .spec file will be included in future releases which should mean you can just use rpmbuild to create the RPM package for you).  The machine I needed to install this onto did not have any compilers install for reasons of security, and so my list of pre-requisites was rather long and included more than I would have liked but needs must, and the compilers were removed afterwards.  But anyway, here is the list of commands issued to get everything where it needed to be;

sudo rpm -ivh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
sudo yum install gcc
sudo yum install openssl-libs openssl-devel
sudo yum install libopendkim opendkim
sudo yum install libtool libidn2 libidn2-devel
sudo yum install opendbx-devel opendbx-mysql opendbx-utils opendbx

Note.  The addition of the elrepo.org rpm is required as there are a couple of dependencies, which don’t appear to be available in the main CentOS yum repository.

Download, compile and install

Now, the original instructions I followed, (see reference material at bottom of this page), took me through the steps of manually compiling the code for and installing opendkim.  After I had installed it, I found that opendkim is available in the CentOS base yum repo and so I could have avoided that step.  You will be pleased to hear that I have not included those steps here (if you do need to know though, see this blog post – strangeworld.com).

Next up we need to obtain the tar ball containing the source code for zdkimfilter, which was downloaded from; http://www.tana.it/sw/zdkimfilter/.  At the time of writing the current version was v1.5.

Now we have that lets work through the process of compiling and installing it.

wget http://www.tana.it/sw/zdkimfilter/zdkimfilter-1.5.tar.gz
tar zxvf zdkimfilter-1.5.tar.gz 
cd zdkimfilter-1.5
./configure --prefix=/usr/local
make -j4
sudo make install

Configuration

Last on the list of installation steps is configuring zdkimfilter and making sure we have the correct directory structure, etc;

cp /etc/courier/filters/zdkimfilter.conf.dist /etc/courier/filters/zdkimfilter.conf
mkdir /etc/courier/filters/keys
chmod u=rwx,g=rw,o-rwx /etc/courier/filters/keys

Nothing more was needed at this time.

Generate some keys and update your DNS zones

At this point, we should have everything installed and configured that we need, and can now proceed with the steps for generating the necessary private keys which will be used to sign our emails as they leave your courier email server and also the public key that will be published in DNS.

A word of warning.  Not sure how important this is, but when looking at generating the keys using opendkim-genkey, I would advise that you choose your selector name carefully.  Initially I set this to be the same as the domain.  I then went to a couple of dkim testing websites and found that one of them didn’t like the “.” (dot) in the selector name.  At this point I changed my plan slightly (in case this was a more wide spread issue) and made sure there were no periods in the selector name.

cd /etc/courier/filters/keys/
opendkim-genkey -b 2048 -d tobyheywood.com -D /etc/courier/filters/keys -s tobyheywood -r --nosubdomains -v
ln -s tobyheywood.private tobyheywood.com
chown root.daemon tobyheywood.*
(Just making sure ownership is correct)
chmod u=rw,g=r,o-rwx tobyheywood.*
(this will change the permissions on the tobyheywood.private file which contains the private key, and the tobyheywood.txt file which contains the public key)

It’s also worth making sure the directory /etc/courier/filters/keys has the right permissions;

chmod u=rwx,g=rx,o-rwx /etc/courier/filters/keys

And also if you are running SELinux, then it is worth double checking the SELinux security context associated with these newly created files.  In my case, I didn’t have to do anything, but best to check.

Now you need to update your DNS zone file(s) with everything in the .txt file up to and including the last “)”.  You could include the rest if you wish, but it’s purely a comment.  Reload you zone and test.

Testing, testing, 1, 2, 3

I performed two types of test.  The first was to make sure the record was returned correctly when queried by third parties and that there were no issues with the DNS side of things.  The website I preferred was; https://www.mail-tester.com/spf-dkim-check as it was a nice and simple site and worked better than the unnamed first site I tried.

The second part of my test, was to send an email to a yahoo or gmail account to confirm that it was accepted and so that I could review the headers.  This turned out to be a very good move!  The headers of my first test emails showed there was no DKIM key present, which I thought was odd, but that was due to the umask and the keys directory didn’t have the execute bit set for group members.  I have corrected this in the above [rough] instructions.

Another attempt at sending an email showed there was a valid key but that the time on this server was ahead by some 6 minutes.  I then found that NTP hadn’t been enabled and so had to enable this in order to get the time aligned with other servers on the Internet.

My last test email, was a complete success.  The headers showed that everything was as it should be.  The DKIM signature had been accepted.

The next thing I need to read up on and implement for the domain in question is dmarc.  But more on that later.

Reference material

Courier MTA and DKIM

Image credit:  Thanks to Jake Rush for uploading the featured image GotCredit to flickr.com.

Firewalld: firewall-cmd example to drop packets from specific ip

Today I spotted some attempts to perform a zone transfer from one of the DNS servers I manage.  Given this is on CentOS 7 and therefore using by default Firewalld, I had a quick read of the documentation regarding how best to drop these attempts.

Here we go;

firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="x.x.x.x" service name="dns" drop'

And that was all that was required.  Note that single quotes are used to contain the entire string.

Should you need some bed time reading, then I would highly recommend reading the following;

http://fedoraproject.org/wiki/Features/FirewalldRichLanguage#firewalld_Rich_Language

Getting started with firewalld

Change is one of those things, that without realising, can begin to get the better of you.  I realised this after I finally decided to take a look at firewalld.  The thought of having to learn a new way to do things when I was already happy with iptables meant that I kept putting it off.

Once you master the basic command options, you are away and it is for the most part self-explanatory.

So here are a few links to get you started;