Red Hat Satellite Server – Fatal error in Python code occurred [[6]]

I have embraced Red Hat Satellite server in a big way over the past year and try to use it wherever possible though not for everything.

One of the features I started using to simply life whilst I look at other configuration management systems, was Configuration Channels.  These allow you to provide a central repository of files and binaries which can be deployed to a server during the initial kickstart server deployment process.

Some changes had been made a month or so ago, to ensure that a specific configuration channel would be included in future deployments by way of updating the Activation Key for that deployment type in Satellite server.  Seems innocent enough at this point.  It is worth noting that there were other configuration channels associated with this activation key.

At the same time I had also added a couple of packages to the software package list which were also required at time of deployment.

Now, I rely on scripts which have been deployed to a server to complete some post server build tasks.  The first thing I noticed after a test deployment, was a complete lack of any scripts where I expected them to be.  The configuration channels had created the required folder structure but had stopped completely and had gone no further.  The error the Satellite server reported back to me was… well not massively helpful;

Fatal error in Python code occurred [[6]]

Nothing more, nothing less.

At this point I started trying to remember what I had added (thankfully not to hard as I document things quite heavily 🙂 ).  Here is roughly the steps I took to confirm whether the issue resided;

  • Remove the additional packages I had specified for this particular build – made no difference
  • Remove what I the most recently added configuration channel – made no difference
  • Tested another Red Hat Enterprise Linux 7 build (not using this particular kickstart profile) – success, so the issue would appear to be limited to this one profile
  • Remove the other configuration channels that were added some time before the last one was added – failed, still the configuration channels would not deploy. But wait, there was light at the end of the tunnel!

But, following this last step, the error message changed, from something not very helpful to something quite helpful indeed!  The message stated that permissions could not be applied as per those stipulated against specific files in the configuration channel.

So it transpires that it was a permissions resolution issue. Well, more a group resolution issue really.  There were a couple of files which were set to be deployed with a specific group.  The group in question is served from a LDAP server and the newly built machine wasn’t configured at that point to talk to the LDAP server, for this particular deployment we didn’t want auto registration with the LDAP services.

So the lesson here is make small changes, test frequently and make sure you document what you have done.  Or use a configuration management system which is version controlled, so you can easily roll back.

Just so we are clear, I was running Red Hat Satellite Server 5.7 (full patched) on RHEL 6.8 and trying to deploy RHEL 7.3.  My adventure to upgrade Satellite server to version 6.2 will be coming to a blog post soon.

So, it would appear this story comes with a lesson attached (free of charge) that all should take note of – “Always make one change at a time and test or as near to one as you can”.

Featured image credit: Charly W Karl posted e.Deorbit closing on target satellite on Flickr.  Thanks very much.

DKIM + Courier-MTA on CentOS7

Email.  One of those things which is critical to business and depending on what you read and who you speak to, you may be led to believe that it is dying out in favour of instant messaging technologies.  Well, as of right now, I can’t see it dying out any time soon, though it does come with a really annoying, frustrating, excruciating problem.  Junk mail!  Otherwise know as spam!  It has once again become a bit of a problem for me, especially when the likes of yahoo begin to imped the flow of emails leaving one of the servers I have worked on recently unable to send emails to recipients with Yahoo accounts.

Now, don’t get my wrong.  I don’t hold a grudge with any of the large email service providers for blocking spam which has sadly managed to be sent via a mail server I have some involvement with, but I do wish the world wasn’t such a bad place that we have to continuously fight on our email frontiers to protect our virtual/real name and brand.

So anyway.  On with the show

The server

  • CentOS 7
  • Patched to the nines
  • Courier MTA
  • Blocking of known spammers was enabled
  • Spamassassin was used as a secondary measure should the hard blocks fail
  • SPF is configured for all hosted domains
  • DKIM was not configured
  • DMAC was not configured
  • It *was not* an open relay
  • The time was not in sync, which  didn’t help later down the line

The problem

The server’s “reputation” had been tarnished as someone had managed to start sending spam via the server through a vulnerability which was found in one of the websites hosted on the server.

After reviewing the advice from Yahoo, I started my investigation on how to approach this topic.

A bit of googling, led me to this web site; https://www.strangeworld.com/blog/archives/128. Now for me, there were a few things missing in the instructions, though complete were it was most needed.  So recreating some of the  good work done on the strangeworld.com website and providing my own twist on things here;

DKIM & ZDKIMFILTER Pre-requisites

In order to install zdkimfilter you need to manually compile the code and install (a .spec file will be included in future releases which should mean you can just use rpmbuild to create the RPM package for you).  The machine I needed to install this onto did not have any compilers install for reasons of security, and so my list of pre-requisites was rather long and included more than I would have liked but needs must, and the compilers were removed afterwards.  But anyway, here is the list of commands issued to get everything where it needed to be;

sudo rpm -ivh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
sudo yum install gcc
sudo yum install openssl-libs openssl-devel
sudo yum install libopendkim opendkim
sudo yum install libtool libidn2 libidn2-devel
sudo yum install opendbx-devel opendbx-mysql opendbx-utils opendbx

Note.  The addition of the elrepo.org rpm is required as there are a couple of dependencies, which don’t appear to be available in the main CentOS yum repository.

Download, compile and install

Now, the original instructions I followed, (see reference material at bottom of this page), took me through the steps of manually compiling the code for and installing opendkim.  After I had installed it, I found that opendkim is available in the CentOS base yum repo and so I could have avoided that step.  You will be pleased to hear that I have not included those steps here (if you do need to know though, see this blog post – strangeworld.com).

Next up we need to obtain the tar ball containing the source code for zdkimfilter, which was downloaded from; http://www.tana.it/sw/zdkimfilter/.  At the time of writing the current version was v1.5.

Now we have that lets work through the process of compiling and installing it.

wget http://www.tana.it/sw/zdkimfilter/zdkimfilter-1.5.tar.gz
tar zxvf zdkimfilter-1.5.tar.gz 
cd zdkimfilter-1.5
./configure --prefix=/usr/local
make -j4
sudo make install

Configuration

Last on the list of installation steps is configuring zdkimfilter and making sure we have the correct directory structure, etc;

cp /etc/courier/filters/zdkimfilter.conf.dist /etc/courier/filters/zdkimfilter.conf
mkdir /etc/courier/filters/keys
chmod u=rwx,g=rw,o-rwx /etc/courier/filters/keys

Nothing more was needed at this time.

Generate some keys and update your DNS zones

At this point, we should have everything installed and configured that we need, and can now proceed with the steps for generating the necessary private keys which will be used to sign our emails as they leave your courier email server and also the public key that will be published in DNS.

A word of warning.  Not sure how important this is, but when looking at generating the keys using opendkim-genkey, I would advise that you choose your selector name carefully.  Initially I set this to be the same as the domain.  I then went to a couple of dkim testing websites and found that one of them didn’t like the “.” (dot) in the separator name.  At this point I changed my plan slightly (in case this was a more wide spread issue) and made sure there were no periods in the separator name.

cd /etc/courier/filters/keys/
opendkim-genkey -b 2048 -d tobyheywood.com -D /etc/courier/filters/keys -s tobyheywood -r -nosubdomains -v
ln -s tobyheywood tobyheywood.com
chown root.daemon tobyheywood.*
(Just making sure ownership is correct)
chmod u=rw,g=r,o-rwx tobyheywood.*
(this will change the permissions on the tobyheywood.private file which contains the private key, and the tobyheywood.txt file which contains the public key)

It’s also worth making sure the directory /etc/courier/filters/keys has the right permissions;

chmod u=rwx,g=rx,o-rwx /etc/courier/filters/keys

And also if you are running SELinux, then it is worth double checking the SELinux security context associated with these newly created files.  In my case, I didn’t have to do anything, but best to check.

Now you need to update your DNS zone file(s) with everything in the .txt file up to the last “)”.  You could include the rest if you wish, but it’s purely a comment.  Reload you zone and test.

Testing, testing, 1, 2, 3

I performed two types of test.  The first was to make sure the record was returned correctly when queried by third parties and that there were no issues with the DNS side of things.  The website I preferred was; https://www.mail-tester.com/spf-dkim-check as it was a nice and simple site and worked better than the unnamed first site I tried.

The second part of my test, was to send an email to a yahoo or gmail account to confirm that it was accepted and so that I could review the headers.  This turned out to be a very good move!  The headers of my first test emails showed there was no DKIM key present, which I thought was odd, but that was due to the umask and the keys directory didn’t have the execute bit set for group members.  I have corrected this in the above [rough] instructions.

Another attempt at sending an email showed there was a valid key but that the time on this server was ahead by some 6 minutes.  I then found that NTP hadn’t been enabled and so had to enable this in order to get the time aligned with other servers on the Internet.

My last test email, was a complete success.  The headers showed that everything was as it should be.  The DKIM signature had been accepted.

The next thing I need to read up on and implement for the domain in question is dmarc.  But more on that later.

Reference material

Courier MTA and DKIM

Image credit:  Thanks to Jake Rush for uploading the featured image GotCredit to flickr.com.

GNS3 – Problems Compiling IOUYAP

Following the series of posts over recent months relating to the installation of GNS3; 1) Installing GNS3 on Fedora, CentOS and RHEL and 2) GNS3 Installation on Fedora 22 I have decided to put together a complete guide from start to finish chronicling my every step to ensure my GNS3 installation is correct and fully working.

It is at this point that I feel that I should not just share the normal story of “everything went fine… what problems?? I didn’t have a single issue installing the software!!!”  Well, for the most part I could have got away with a statement like that.  Unfortunately though when I reached the point of installing iouyap, things didn’t go quite as smoothly as one would hope.

As per my previous guides and the instructions that ship with iouyap (version 0.95) I ran the following;

$ bison --yacc -dv netmap_parse.y
$ flex netmap_scan.l
$ gcc -Wall -g *.c -o iouyap -liniparser -lpthread

I was the confronted with a wall of potential errors, and not knowing exactly what they meant, I thought it best to investigate…  The output from the last command line above was;

[toby@thebay iouyap-0.95]$ sudo gcc -Wall -g *.c -o iouyap -liniparser -lpthread
In file included from iouyap.c:49:0:
iouyap.c: In function ‘foreign_listener’:
iouyap.c:420:24: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘ssize_t {aka long int}’ [-Wformat=]
debug_log_fmt ("received %d bytes (sfd=%d)\n",
^
iouyap.h:103:28: note: in definition of macro ‘STMT’
#define STMT( code )  do { code } while (0)
^
iouyap.h:128:5: note: in expansion of macro ‘STMT’
STMT( log_prefix(); fprintf(stderr, fmt, ## __VA_ARGS__); )
^
iouyap.h:138:26: note: in expansion of macro ‘log_fmt’
STMT( if (DEBUG_LOG) log_fmt(fmt, ## __VA_ARGS__); )
^
iouyap.c:420:9: note: in expansion of macro ‘debug_log_fmt’
debug_log_fmt ("received %d bytes (sfd=%d)\n",
^
iouyap.c:458:28: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘ssize_t {aka long int}’ [-Wformat=]
log_fmt ("sendto() only sent %d of %d bytes!"
^
iouyap.h:103:28: note: in definition of macro ‘STMT’
#define STMT( code )  do { code } while (0)
^
iouyap.c:458:19: note: in expansion of macro ‘log_fmt’
log_fmt ("sendto() only sent %d of %d bytes!"
^
iouyap.c:458:28: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has type ‘ssize_t {aka long int}’ [-Wformat=]
log_fmt ("sendto() only sent %d of %d bytes!"
^
iouyap.h:103:28: note: in definition of macro ‘STMT’
#define STMT( code )  do { code } while (0)
^
iouyap.c:458:19: note: in expansion of macro ‘log_fmt’
log_fmt ("sendto() only sent %d of %d bytes!"
^
iouyap.c: In function ‘iou_listener’:
iouyap.c:536:24: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘ssize_t {aka long int}’ [-Wformat=]
debug_log_fmt ("received %d bytes for port %d (sfd=%d)\n",
^
iouyap.h:103:28: note: in definition of macro ‘STMT’
#define STMT( code )  do { code } while (0)
^
iouyap.h:128:5: note: in expansion of macro ‘STMT’
STMT( log_prefix(); fprintf(stderr, fmt, ## __VA_ARGS__); )
^
iouyap.h:138:26: note: in expansion of macro ‘log_fmt’
STMT( if (DEBUG_LOG) log_fmt(fmt, ## __VA_ARGS__); )
^
iouyap.c:536:9: note: in expansion of macro ‘debug_log_fmt’
debug_log_fmt ("received %d bytes for port %d (sfd=%d)\n",
^
iouyap.c:563:24: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘ssize_t {aka long int}’ [-Wformat=]
log_fmt ("write() only sent %d of %d bytes! (sfd=%d)\n",
^
iouyap.h:103:28: note: in definition of macro ‘STMT’
#define STMT( code )  do { code } while (0)
^
iouyap.c:563:15: note: in expansion of macro ‘log_fmt’
log_fmt ("write() only sent %d of %d bytes! (sfd=%d)\n",
^
iouyap.c:563:24: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has type ‘ssize_t {aka long int}’ [-Wformat=]
log_fmt ("write() only sent %d of %d bytes! (sfd=%d)\n",
^
iouyap.h:103:28: note: in definition of macro ‘STMT’
#define STMT( code )  do { code } while (0)
^
iouyap.c:563:15: note: in expansion of macro ‘log_fmt’
log_fmt ("write() only sent %d of %d bytes! (sfd=%d)\n",
^
In file included from netmap.c:34:0:
netmap.c: In function ‘dump_port_table’:
netmap.c:372:16: warning: format ‘%d’ expects argument of type ‘int’, but argument 6 has type ‘size_t {aka long unsigned int}’ [-Wformat=]
log_fmt ("%d:%d/%d talks to %d other node(s):\n", yap_appl_id,
^
iouyap.h:103:28: note: in definition of macro ‘STMT’
#define STMT( code )  do { code } while (0)
^
netmap.c:372:7: note: in expansion of macro ‘log_fmt’
log_fmt ("%d:%d/%d talks to %d other node(s):\n", yap_appl_id,
^

After some monkeying about, I reduced these errors to;

In file included from netmap.c:34:0:
netmap.c: In function ‘dump_port_table’:
netmap.c:372:16: warning: format ‘%d’ expects argument of type ‘int’, but argument 6 has type ‘size_t {aka long unsigned int}’ [-Wformat=]
log_fmt ("%d:%d/%d talks to %d other node(s):\n", yap_appl_id,
^
iouyap.h:103:28: note: in definition of macro ‘STMT’
#define STMT( code )  do { code } while (0)
^
netmap.c:372:7: note: in expansion of macro ‘log_fmt’
log_fmt ("%d:%d/%d talks to %d other node(s):\n", yap_appl_id,
^

the diff output is as follows;

diff iouyap.c ../../iouyap-0.95_original/iouyap-0.95/iouyap.c
383,384c383
<   /* ssize_t bytes_received, bytes_sent; */
<   int bytes_received, bytes_sent; --- >   ssize_t bytes_received, bytes_sent;
503,504c502
<   /* ssize_t bytes_received, bytes_sent; */
<   int bytes_received, bytes_sent; --- >   ssize_t bytes_received, bytes_sent;

Now I’m no C++ developer, not even close, and the last thing to resolve in netmap.c just didn’t make sense. So I paid the gns3 website a visit, and magically less than 24 hours after I had downloaded the 1.3.8 version there was a new release… 1.3.9.

Sadly the IOUYAP package there also contained the issue, in the end I resorted to good old fashioned tactics… I got hold of the source code from github and the issues I was facing were fixed in the version I checked out.  You can see the results below;

[toby@thebay GNS3-1.3.9]$ git clone https://github.com/GNS3/iouyap.git
Cloning into 'iouyap'...
remote: Counting objects: 78, done.
remote: Total 78 (delta 0), reused 0 (delta 0), pack-reused 78
Unpacking objects: 100% (78/78), done.
Checking connectivity... done.
[toby@thebay GNS3-1.3.9]$ ls
dynamips-0.2.14      gns3-gui-1.3.9      gns3-server-1.3.9      iouyap       iouyap-0.95.zip  ubridge-0.9.0.zip  vpcs-0.6.1.zip
dynamips-0.2.14.zip  gns3-gui-1.3.9.zip  gns3-server-1.3.9.zip  iouyap-0.95  ubridge-0.9.0    vpcs-0.6.1
[toby@thebay GNS3-1.3.9]$ cd iouyap
[toby@thebay iouyap]$ ls
config.c  dictionary.h  iouyap.c  iouyap.ini  Makefile  netmap.c  netmap_parse.y  README.rst
config.h  iniparser.h   iouyap.h  LICENSE     NETMAP    netmap.h  netmap_scan.l
[toby@thebay iouyap]$ make
gcc  -g -DDEBUG -Wall   -c -o iouyap.o iouyap.c
bison -y -d netmap_parse.y
mv -f y.tab.c netmap_parse.c
gcc  -g -DDEBUG -Wall   -c -o netmap_parse.o netmap_parse.c
flex  -t netmap_scan.l > netmap_scan.c
gcc  -g -DDEBUG -Wall   -c -o netmap_scan.o netmap_scan.c
gcc  -g -DDEBUG -Wall   -c -o netmap.o netmap.c
gcc  -g -DDEBUG -Wall   -c -o config.o config.c
gcc    iouyap.o netmap_parse.o netmap_scan.o netmap.o config.o  -liniparser -lpthread -o iouyap
rm netmap_scan.c netmap_parse.c
[toby@thebay iouyap]$ sudo make install
[sudo] password for toby:
chmod +x iouyap
sudo cp iouyap /usr/local/bin
sudo setcap cap_net_admin,cap_net_raw=ep iouyap

Taa Daa!