I have embraced Red Hat Satellite server in a big way over the past year and try to use it wherever possible though not for everything.
One of the features I started using to simply life whilst I look at other configuration management systems, was Configuration Channels. These allow you to provide a central repository of files and binaries which can be deployed to a server during the initial kickstart server deployment process.
Some changes had been made a month or so ago, to ensure that a specific configuration channel would be included in future deployments by way of updating the Activation Key for that deployment type in Satellite server. Seems innocent enough at this point. It is worth noting that there were other configuration channels associated with this activation key.
At the same time I had also added a couple of packages to the software package list which were also required at time of deployment.
Now, I rely on scripts which have been deployed to a server to complete some post server build tasks. The first thing I noticed after a test deployment, was a complete lack of any scripts where I expected them to be. The configuration channels had created the required folder structure but had stopped completely and had gone no further. The error the Satellite server reported back to me was… well not massively helpful;
Fatal error in Python code occurred []
Nothing more, nothing less.
At this point I started trying to remember what I had added (thankfully not to hard as I document things quite heavily 🙂 ). Here is roughly the steps I took to confirm whether the issue resided;
- Remove the additional packages I had specified for this particular build – made no difference
- Remove what I the most recently added configuration channel – made no difference
- Tested another Red Hat Enterprise Linux 7 build (not using this particular kickstart profile) – success, so the issue would appear to be limited to this one profile
- Remove the other configuration channels that were added some time before the last one was added – failed, still the configuration channels would not deploy. But wait, there was light at the end of the tunnel!
But, following this last step, the error message changed, from something not very helpful to something quite helpful indeed! The message stated that permissions could not be applied as per those stipulated against specific files in the configuration channel.
So it transpires that it was a permissions resolution issue. Well, more a group resolution issue really. There were a couple of files which were set to be deployed with a specific group. The group in question is served from a LDAP server and the newly built machine wasn’t configured at that point to talk to the LDAP server, for this particular deployment we didn’t want auto registration with the LDAP services.
So the lesson here is make small changes, test frequently and make sure you document what you have done. Or use a configuration management system which is version controlled, so you can easily roll back.
Just so we are clear, I was running Red Hat Satellite Server 5.7 (full patched) on RHEL 6.8 and trying to deploy RHEL 7.3. My adventure to upgrade Satellite server to version 6.2 will be coming to a blog post soon.
So, it would appear this story comes with a lesson attached (free of charge) that all should take note of – “Always make one change at a time and test or as near to one as you can”.
Featured image credit: Charly W Karl posted e.Deorbit closing on target satellite on Flickr. Thanks very much.