I have embraced Red Hat Satellite server in a big way over the past year and try to use it wherever possible though not for everything.
One of the features I started using to simply life whilst I look at other configuration management systems, was Configuration Channels. These allow you to provide a central repository of files and binaries which can be deployed to a server during the initial kickstart server deployment process.
Some changes had been made a month or so ago, to ensure that a specific configuration channel would be included in future deployments by way of updating the Activation Key for that deployment type in Satellite server. Seems innocent enough at this point. It is worth noting that there were other configuration channels associated with this activation key.
At the same time I had also added a couple of packages to the software package list which were also required at time of deployment.
Now, I rely on scripts which have been deployed to a server to complete some post server build tasks. The first thing I noticed after a test deployment, was a complete lack of any scripts where I expected them to be. The configuration channels had created the required folder structure but had stopped completely and had gone no further. The error the Satellite server reported back to me was… well not massively helpful;
Fatal error in Python code occurred []
Nothing more, nothing less.
At this point I started trying to remember what I had added (thankfully not to hard as I document things quite heavily 🙂 ). Here is roughly the steps I took to confirm whether the issue resided;
Remove the additional packages I had specified for this particular build – made no difference
Remove what I the most recently added configuration channel – made no difference
Tested another Red Hat Enterprise Linux 7 build (not using this particular kickstart profile) – success, so the issue would appear to be limited to this one profile
Remove the other configuration channels that were added some time before the last one was added – failed, still the configuration channels would not deploy.But wait, there was light at the end of the tunnel!
But, following this last step, the error message changed, from something not very helpful to something quite helpful indeed! The message stated that permissions could not be applied as per those stipulated against specific files in the configuration channel.
So it transpires that it was a permissions resolution issue. Well, more a group resolution issue really. There were a couple of files which were set to be deployed with a specific group. The group in question is served from a LDAP server and the newly built machine wasn’t configured at that point to talk to the LDAP server, for this particular deployment we didn’t want auto registration with the LDAP services.
So the lesson here is make small changes, test frequently and make sure you document what you have done. Or use a configuration management system which is version controlled, so you can easily roll back.
Just so we are clear, I was running Red Hat Satellite Server 5.7 (full patched) on RHEL 6.8 and trying to deploy RHEL 7.3. My adventure to upgrade Satellite server to version 6.2 will be coming to a blog post soon.
So, it would appear this story comes with a lesson attached (free of charge) that all should take note of – “Always make one change at a time and test or as near to one as you can”.
Spacewalk utilises a database back end to store the required information about your environment. The two options are PostgreSQL and Oracle. Neither would be my preference but I always opt for the lesser of two evils – PostgreSQL.
The installation is a piece of cake, and can be performed by issuing the following command at the command line;
yum install spacewalk-setup-postgresql -y
During the process you should be prompted to accept the Spacewalk GPG key. You will need to enter “y” to accept!
Now things have been made pretty easy for you so far. And we wont stop now. To install all of the required packages for spacewalk just run the following;
yum install spacewalk-postgresql
And let it download everything you need. In all (at the time of writing) there were 379 packages totalling 563M.
Again you will likely be prompted to import the Fedora EPEL (7) GPG key. This is necessary so just type “y” and give that Enter key a gentle tap.
And.. you will also be prompted to import the JPackage Project GPG key. Same process as above – “y” followed by Enter.
During the installation you will see a lot of text scrolling up the screen. This will be a mix of general package installation output from yum and some commands that the RPM package will initiate to set and define such things as SELinux contexts.
The key thing is you should see right at the end “Complete!”. You know you are in a good place at this point.
Security: Setting up the firewall rules
CentOS 7 and (for that matter) Red Hat Enterprise Linux 7 ship with firewalld as standard. Now I’m not complete sure of firewalld but I’m sticking with it, but should you decide you want to use iptables (and you have taken steps to make sure it is enabled), then I have provided the firewall rules required for both;
Note. Make sure you have double dashes/hyphens if you copy and paste as I have seen the pasted text only using a single hyphen.
Skip to section after iptables if you have applied the above configuration!
Now as iptables can be configured in all manor or ways, I’m just going to provide the basics, if your set-up is typically more customised than the default, then you probably don’t need me telling you how to setup iptables.
I will just make one assumption though. That the default INPUT policy is set to DROP and than you do not have any DROP, REJECT lines at the end of your INPUT chain.
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
And don’t forget to save your firewall rules;
# service iptables save
Right then, still with me? Awesome, so lets continue with getting Spacewalk up and running. At this point there is one fundamental thing you need…
You must have a resolvable Fully Qualified Domain Name (FQDN). For my installation I have fudged it and added the FQDN to the host file, as I intend to build the rest of my new lab environment using Spacewalk.
So assuming you have followed everything above we can now simply run the following;
Note.The above assumes you have the embedded PostgreSQL database and not a remote DB, or the Oracle DB option. Just saying.
So you should see something like the following (it may take quite some time for many of the tasks to be completed so bare with it);
[root@spacewalk ~]# spacewalk-setup
* Setting up SELinux..
** Database: Setting up database connection for PostgreSQL backend.
** Database: Installing the database:
** Database: This is a long process that is logged in:
** Database: /var/log/rhn/install_db.log
*** Progress: ###
** Database: Installation complete.
** Database: Populating database.
*** Progress: ###########################
* Configuring tomcat.
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
** GPG: Creating /root/.gnupg directory
You must enter an email address.
Admin Email Address? firstname.lastname@example.org
* Performing initial configuration.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]?
** /etc/httpd/conf.d/ssl.conf has been backed up to ssl.conf-swsave
* Configuring jabberd.
* Creating SSL certificates.
CA certificate password?
Re-enter CA certificate password?
Organization? Toby Heywood
Organization Unit [spacewalk]?
Email Address [email@example.com]?
Country code (Examples: "US", "JP", "IN", or type "?" to see a list)? GB
** SSL: Generating CA certificate.
** SSL: Deploying CA certificate.
** SSL: Generating server certificate.
** SSL: Storing SSL certificates.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? y
* Restarting services.
Visit https://spacewalk to create the Spacewalk administrator account.
Now at this point you are almost ready to break open a beer and give yourself a pat on the back. But lets finalise the installation first.
Creating your Organisation
(that’s Organization for the Americans)
Setting up your organisation requires only a few simple things to be provided.
Click the Create Organization button and you should finally see a similar screen to the following;
The last thing to do now you have your shiny new installation of Spacewalk is to perform a few sanity checks;
Navigate to Admin > Task Engine Status and confirm that everything looks health and that the Scheduling Service is showing as “ON”
You can also take a look at my earlier blog post – spacewalk sanity checking – about some steps I previously took to make sure everything was running.
When it comes to setting up Spacewalk to provide and meet you organisations package management and provisioning needs, there is more to it than simply installing the Spacewalk and then clicking Provision! There is a list of hoops to jump through before you can get up and running. This post aims to tackle the common setup tasks through to your first client registration, but specifically with respect to installing it on a system running CentOS 7. But lets not get ahead of ourselves. There is a lot to do, so lets get cracking!
I am assuming here that you have managed to install Spacewalk and are now looking for the next steps starting at creating the administrative user. If not may I suggest taking a peek here as I have provided a very rough guide to do this. I am also basing a lot of these steps on the HowTo published on the CentOS wiki, though that was for release 5 of CentOS, not 7. So I will try to fill in gaps where required.
Initial configuration of Spacewalk
OK, you have at this point, hopefully got Spacewalk/Satellite server installed. The first thing to do at this point is to login to the web GUI. Well, when I say login, I mean create the admin account. Best to do this right away before someone has the opportunity to takeover your nice new Spacewalk/Red Hat Satellite server. You access the GUI by typing in the FQDN of the spacewalk server and it will redirect you to the Create Spacewalk Administrator. You should see a screen much like the following;
Just enter a few essential details and away you go. OK you won’t get too far yet, but keep reading!
Upon clicking the “Create Login” button, you should see the normal dashboard screen that is displayed when logging into Spacewalk (and Red Hat Satellite) for the first time. With one exception. You should also have a banner across the top with the following wording;
You have created your first user for the Spacewalk Service. Additional configuration should be finalized by Click here
Make sure you click the “Click here” link and make sure you complete the rest of the steps.
I would advise you check and double check the General configuration tab, specifically the Spacewalk hostname, this should ideally match the FQDN of your satellite server. And if you haven’t specified a name which is resolvable via DNS you will likely find that things don’t run exactly as they should.
The Certificate tab will be of interest if you are minting your own SSL certificates or wish to use a commercially generated cert. The Bootstrap script tab is where you define settings relating to how clients connect and the associated security around those connections.
The Organizations tab (which in my opinion should read Organisations (because that’s how you spell it) is where you can define how your organisation looks, you can define multiple activation keys for different parts of your organisation, manage subscriptions and users. To name but a few of the things you can do.
Restart tab, er, do I really need to suggest what this does??? And finally the Cobbler tab. From here you can kick of a synchronisation between Spacewalk and Cobblerd. I recommend clicking it now to make sure the integration between the two applications is working. I would also suggest you double check the cobbler log file located at /var/log/cobbler/cobbler.log for any signs of problems. Here’s a sample output;
In order to register systems against your newly installed Spacewalk server, you must have a activation key defined. This is not done automatically, and therefore we shall tackle it now. Navigate to Systems > Activation Keys.
Initially you should see a message stating;
You do not currently have a universal default activation key set. To set a key as the universal default, please visit the details page of that key and check off the 'Universal Default?' checkbox.
Click + Create Key in the top right hand corner of the Activation Keys screen. You will need to add the following details;
A Key (I would advise putting something meaningful in here, rather than allowing a key to be auto-generated.
Leave the Usage field blank
Leave the Base Channels as default (Spacewalk Default)
Add-on Entitlements, I have selected only Provisioning (it can be changed later)
I also ticked the Universal Default as I do not want to restrict its use
After the default key has been created, the screen looks like this;
Creating your first package repository (and channel)
I will be focusing on CentOS 7 here, but Satellite is capable of providing a centralised repository for other RPM based distributions.
CentOS 7 Base Repository
We will assume you may at some point want to build further servers using the base OS RPMs. The first thing you need to do is find a local mirror site which you can base your repository on. CentOS provide a lovely page on their web site – https://www.centos.org/download/mirrors/, which details, (by country) where you can download the packages from. In my case I searched the page for United Kingdom and picked one from the list.
Lets get on with the job at hand, and create a repository. Click Channels > Manage Software Channels > Manage Repositories. And then click Create Repository. You will then see a screen, not too dissimilar to the one below;
Once you have defined the repository label and URL (this is the source url which Spacewalk will be using to obtain the packages from). I have defined the SSL cert that was generated during installation.
You will now need to create the Channel that will be associated to this channel. Click Manage Software Channels from the left hand menu and then click on Create Channel. You will (once the page has loaded) be given a few ground rules regarding naming conventions and then the opportunity to create your new channel. The long and short of it is this;
Channel Name and Channel Label are required (hence the red asterisk)
must be between 6 and 256 characters in length
must begin with a letter
may contain spaces, parentheses () and forward slashes /.
Channel Label must;
be no longer than 128 characters
start with a letter or digit
Must be lowercase (no exceptions)
May contain hyphens, periods, underscores and numerals
Some other options on the screen also include controlling access to the repository (i.e. is it private and only accessible to your Spacewalk organisation, or is it public), also you can define GPG security settings for signed packages.
The last step is to marry the repository and channel together. This is achieved by going to the Repositories tab, and selecting the repository from the list of available repositories. In my case it is just one.
The last step, will be kick off a synchronisation of the repository. Now there are two ways to do this; 1) By click the Sync tab, then ticking the “Create kickstartable tree” option and then clicking Sync Now. Or 2) run the following command from the cli.
Now sit back and watch/wait. For the current 7.2 repo, the base number of packages is just over 9000, so depending on your connection to the internet, you could find this to be a quick process or quite slow (also very dependent upon the mirror you have selected). Another option which I haven’t don’t but I believe it would work, is to use a copy of the installation media. If you try that option, let me know how you get on 🙂
Registering your first client
First, you will need to make sure you have the required packages on the client to be register. In my case, I had used a minimal install and as such I was missing the required packages. Easily rectified;
[root@rhc-client ~]# yum install rhn-setup
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
--> Running transaction check
---> Package rhn-setup.noarch 0:2.0.2-6.el7 will be installed
--> Processing Dependency: rhn-client-tools = 2.0.2-6.el7 for package: rhn-setup-2.0.2-6.el7.noarch
--> Processing Dependency: rhnsd for package: rhn-setup-2.0.2-6.el7.noarch
--> Running transaction check
---> Package rhn-client-tools.noarch 0:2.0.2-6.el7 will be installed
--> Processing Dependency: rhnlib >= 2.5.57 for package: rhn-client-tools-2.0.2-6.el7.noarch
--> Processing Dependency: python-hwdata for package: rhn-client-tools-2.0.2-6.el7.noarch
--> Processing Dependency: python-gudev for package: rhn-client-tools-2.0.2-6.el7.noarch
--> Processing Dependency: python-dmidecode for package: rhn-client-tools-2.0.2-6.el7.noarch
---> Package rhnsd.x86_64 0:5.0.13-5.el7 will be installed
--> Processing Dependency: rhn-check >= 0.0.8 for package: rhnsd-5.0.13-5.el7.x86_64
--> Running transaction check
---> Package python-dmidecode.x86_64 0:3.10.13-11.el7 will be installed
---> Package python-gudev.x86_64 0:147.2-7.el7 will be installed
---> Package python-hwdata.noarch 0:1.7.3-4.el7 will be installed
---> Package rhn-check.noarch 0:2.0.2-6.el7 will be installed
--> Processing Dependency: yum-rhn-plugin >= 1.6.4-1 for package: rhn-check-2.0.2-6.el7.noarch
---> Package rhnlib.noarch 0:2.5.65-2.el7 will be installed
--> Running transaction check
---> Package yum-rhn-plugin.noarch 0:2.0.1-5.el7 will be installed
--> Processing Dependency: m2crypto >= 0.16-6 for package: yum-rhn-plugin-2.0.1-5.el7.noarch
--> Running transaction check
---> Package m2crypto.x86_64 0:0.21.1-17.el7 will be installed
--> Finished Dependency Resolution
Package Arch Version Repository Size
rhn-setup noarch 2.0.2-6.el7 th_lab_server 87 k
Installing for dependencies:
m2crypto x86_64 0.21.1-17.el7 th_lab_server 429 k
python-dmidecode x86_64 3.10.13-11.el7 th_lab_server 82 k
python-gudev x86_64 147.2-7.el7 th_lab_server 18 k
python-hwdata noarch 1.7.3-4.el7 th_lab_server 32 k
rhn-check noarch 2.0.2-6.el7 th_lab_server 52 k
rhn-client-tools noarch 2.0.2-6.el7 th_lab_server 379 k
rhnlib noarch 2.5.65-2.el7 th_lab_server 65 k
rhnsd x86_64 5.0.13-5.el7 th_lab_server 48 k
yum-rhn-plugin noarch 2.0.1-5.el7 th_lab_server 80 k
Install 1 Package (+9 Dependent packages)
Total size: 1.2 M
Total download size: 1.2 M
Installed size: 4.8 M
Is this ok [y/d/N]: y
(1/9): python-gudev-147.2-7.el7.x86_64.rpm | 18 kB 00:00
(2/9): m2crypto-0.21.1-17.el7.x86_64.rpm | 429 kB 00:00
(3/9): python-hwdata-1.7.3-4.el7.noarch.rpm | 32 kB 00:00
(4/9): rhn-check-2.0.2-6.el7.noarch.rpm | 52 kB 00:00
(5/9): rhn-client-tools-2.0.2-6.el7.noarch.rpm | 379 kB 00:00
(6/9): rhn-setup-2.0.2-6.el7.noarch.rpm | 87 kB 00:00
(7/9): rhnlib-2.5.65-2.el7.noarch.rpm | 65 kB 00:00
(8/9): rhnsd-5.0.13-5.el7.x86_64.rpm | 48 kB 00:00
(9/9): yum-rhn-plugin-2.0.1-5.el7.noarch.rpm | 80 kB 00:00
Total 1.3 MB/s | 1.2 MB 00:00
Running transaction check
Running transaction test
Transaction test succeeded
Installing : python-gudev-147.2-7.el7.x86_64 1/10
Installing : rhnlib-2.5.65-2.el7.noarch 2/10
Installing : python-hwdata-1.7.3-4.el7.noarch 3/10
Installing : python-dmidecode-3.10.13-11.el7.x86_64 4/10
Installing : rhn-client-tools-2.0.2-6.el7.noarch 5/10
Installing : m2crypto-0.21.1-17.el7.x86_64 6/10
Installing : rhnsd-5.0.13-5.el7.x86_64 7/10
Installing : rhn-setup-2.0.2-6.el7.noarch 8/10
Installing : yum-rhn-plugin-2.0.1-5.el7.noarch 9/10
Installing : rhn-check-2.0.2-6.el7.noarch 10/10
Verifying : rhn-setup-2.0.2-6.el7.noarch 1/10
Verifying : m2crypto-0.21.1-17.el7.x86_64 2/10
Verifying : rhn-check-2.0.2-6.el7.noarch 3/10
Verifying : python-dmidecode-3.10.13-11.el7.x86_64 4/10
Verifying : rhnsd-5.0.13-5.el7.x86_64 5/10
Verifying : rhn-client-tools-2.0.2-6.el7.noarch 6/10
Verifying : python-hwdata-1.7.3-4.el7.noarch 7/10
Verifying : yum-rhn-plugin-2.0.1-5.el7.noarch 8/10
Verifying : rhnlib-2.5.65-2.el7.noarch 9/10
Verifying : python-gudev-147.2-7.el7.x86_64 10/10
m2crypto.x86_64 0:0.21.1-17.el7 python-dmidecode.x86_64 0:3.10.13-11.el7
python-gudev.x86_64 0:147.2-7.el7 python-hwdata.noarch 0:1.7.3-4.el7
rhn-check.noarch 0:2.0.2-6.el7 rhn-client-tools.noarch 0:2.0.2-6.el7
rhnlib.noarch 0:2.5.65-2.el7 rhnsd.x86_64 0:5.0.13-5.el7
Next step is to install your Spacewalk server’s ssl certificate on the client. This is a security measure which enables the client to verify that the server it is talking to is really the server it SHOULD be talking too.
The final step in the process is to actually register the client against the Spacewalk/Satellite server.
[toby@devops ~]$ sudo rhnreg_ks --serverUrl=https://manager.lab.tobyheywood.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=1-lab.tobyheywood.com
This system is not subscribed to any channels.
RHN channel support will be disabled.
At this point, we float over to the Spacewalk server UI, and we should now see our client in the list of Systems;
Now, for those of you who have a keen eye for detail, you will have noticed in the screenshot above and the snippet of command line output at the time of registration, that the system isn’t currently subscribed to any channels. This is very easily remedied;
Click on the client in the system list
On the initial Overview screen, you will see a box – Subscribed Channels
Click Alter Channel Subscriptions
Now Select from the list under Base Software Channel – in my case “centos_7_base”
You should now see the channel listed under the heading “Software Channel Subscriptions”.
In addition you may have child channels created beneath your base channel.
And there we have it. Time to have a play and see what you can do by having a click around the tabs related to the system and the wider Spacewalk UI.
Following on from an earlier post, it would seem that the “Warning /dev/root does not exist” issue is not confined to “none” kickstart pxe booted installations as I had first thought.
I was working on a RHEL 7 installation using Red Hat Satellite 5.7 (upgrade to 6.x in the pipeline but bigger fish to fry right now), where we were re-using a lot of the RHEL 6 pxelinux kernel parameter’s.
Now as you may or may not know (if you have read my other posts on the topic), there are numerous Anaconda and dracut parameter’s that can be passed to the kernel in the pxelinux.cfg/default (or the mac specific) config file. The problem we had found was the existence off a ksdevice= parameter which pointed to eth0. In RHEL/CentOS 7, the ethernet device naming standard changes from ethX to ensX, which works out as follows;
en = Ethernet
sX = Slot X (where X is the physical or virtual slot number where the nic resides)
By default the first interface is used by anaconda/dracut/pxelinux, IF no option is specified. If however you specifically tell it to use something which fundamentally doesn’t exist, it WILL still try to use that… and fail! Miserably! And give you an error which ultimately seems kind of unrelated.
You have been warned!
As with many things in life, this served as a reminder that things change and that you can’t always take the “old” and reuse with the “new” without issue.
After having installed Spacewalk, got it working to a certain point and then found that there may have been issues with the installation, I thought it would be easier to simply re-install spacewalk onto a new virtual machine.
So following on from my how to article, I wanted to make sure that post installation, I had performed sufficient checks to confirm that there were no issues with the scheduler service or cobbler, as these were two things I had great difficulty trying to get working.
I guess it is also worth mentioning that the VM I am running spacewalk on has a single vCPU and 4GB of memory. For storage I have given it 40G which will do me fine. And as for the OS it is running CentOS 7 (1511).
So what should we check
Good question. The following is a rough list of all the services I confirmed as enabled, running and also that there were no horrible errors in the log files
taskomatic (a.k.a. the scheduler)
[toby@manager ~]$ sudo systemctl status cobblerd
● cobblerd.service - LSB: daemon for libvirt virtualization API
Loaded: loaded (/etc/rc.d/init.d/cobblerd)
Active: active (running) since Fri 2016-04-15 23:08:37 BST; 24min ago
└─13257 /usr/bin/python -s /bin/cobblerd --daemonize
Apr 15 23:08:35 manager systemd: Starting LSB: daemon for libvirt virtualization API...
Apr 15 23:08:37 manager cobblerd: Starting cobbler daemon: [ OK ]
Apr 15 23:08:37 manager systemd: Started LSB: daemon for libvirt virtualization API.
Apr 15 23:31:35 manager systemd: [/run/systemd/generator.late/cobblerd.service:8] Failed to add dependency on network,.service, ignoring: Invalid argument
Apr 15 23:31:35 manager systemd: [/run/systemd/generator.late/cobblerd.service:8] Failed to add dependency on xinetd,.service, ignoring: Invalid argument
The last two lines can be ignore. I believe this is purely due some references to the sysvinit scripts which are no longer used, and as you will see later things appear to be running fine (this time around)
Now, this one doesn’t appear to have been moved over to the new systemd environment and therefore we resort back to the good old sysvinit scripts and the service command to confirm this one is working;
[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (13296).
Looking good so far
But can it withstand a reboot? Now that is the question. So I repeated the above steps again, just to confirm. I won’t bore you with all the details;
cobblerd.service – active (running)
httpd.service – active (running)
tftp.service – inactive (dead)
postgresql.service – active (running)
tomcat.service –active (running)
taskomatic – RHN Taskomatic is not running.
[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is not running.
[toby@manager ~]$ sudo chkconfig taskomatic on
[toby@manager ~]$ sudo service taskomatic start
Starting RHN Taskomatic...
[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (10870).
And for good measure I gave the machine another reboot, just to confirm the taskomatic service did start.
[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (1278).
Oh Yeah! Now I’m a happy camper. And it’s time to re-visit the initial configuration part, which I shall post about shortly.
Image credit; Thanks to Mark Walsh for making the featured image called “Russell Street Court Cells – Padded Cell” available on Flickr.com.