What will happen to the .UK Domain Names after Brexit

Posted on Leave a commentPosted in Uncategorized

Just a thought… Following the EU referendum and the subsequent rift which was highlighted by the very close vote, plus the protests that have taken place since, but just maybe, the United Kingdom is not necessarily as united as we may hope.

Hopefully the new government led by prime minister Theresa May will pull it out of the bag and be able to rebuild and reunify the countries that make up the UK, only time will tell!

Currently the UK enjoys the use of the .uk domain space.  If the UK were to fracture further, at what point will/could .uk become surplus to requirements (if the worst happened)?  I guess the simply answer is when we are simply no longer united and the government has agreed to relinquish further control.

Would the .gb domain name need to be re-instated, which though still online, no one can register new domains.  It is managed by an organisation called JISC which provides much of the academic IT infrastructure used by UK schools, colleges and maybe even university (I didn’t do that much research in all honesty).

I wounder if the there are already plans in the pipeline?  Also how far along are they?

Just think though, the opportunities are there to commercialise the .gb domain.  Had it been available, maybe Great Britain’s Athletes would have bragged about the whopping medal collect following Rio 2016, they could have had the domain team.gb?

Anyway.  Hopefully that will not happen.

Featured image: London telephone box was taken buy Gordon and is available here; https://farm8.staticflickr.com/7684/27845667002_30995290be_k.jpg

kernel: BUG: soft lockup – CPU#0 stuck for 67s!

Posted on Leave a commentPosted in Kernel Tuning, Linux, RHEL 6, System Administration

Over the weekend I was confronted by the above error being repeated on the console of a VM running Oracle RDBMS.

This error occurs when there is a shortage of CPU resources.  For me the solution was a quick shut down of the VM and increasing the available CPU resources.  However there are more ways to skin this cat…

There is also a kernel parameter which can be tweaked;

kernel.softlockup_thresh=x

Were “x” is the threshold (in seconds) you want to allow the kernel to wait before decided there has been a soft lockup.

The Red Hat documentation showed a threshold of 30 seconds.  So I would recommend a bit of experimentation if you feel that 30 seconds is not high enough.  Or throw more resources at it.

Reference Material

Featured image: The Old Lockup was made available by John Powell on Flickr.

Spacewalk – Initial configuration and registering your first client (on CentOS 7)

Posted on Leave a commentPosted in CentOS 7, Deployment, Linux, RHEL 7, RPM, System Administration, Systems Management

When it comes to setting up Spacewalk to provide and meet you organisations package management and provisioning needs, there is more to it than simply installing the Spacewalk and then clicking Provision!  There is a list of hoops to jump through before you can get up and running.  This post aims to tackle the common setup tasks through to your first client registration, but specifically with respect to installing it on a system running CentOS 7.  But lets not get ahead of ourselves.  There is a lot to do, so lets get cracking!

I am assuming here that you have managed to install Spacewalk and are now looking for the next steps starting at creating the administrative user.  If not may I suggest taking a peek here as I have provided a very rough guide to do this.  I am also basing a lot of these steps on the HowTo published on the CentOS wiki, though that was for release 5 of CentOS, not 7.  So I will try to fill in gaps where required.

Initial configuration of Spacewalk

OK, you have at this point, hopefully got Spacewalk/Satellite server installed.  The first thing to do at this point is to login to the web GUI.  Well, when I say login, I mean create the admin account.  Best to do this right away before someone has the opportunity to takeover your nice new Spacewalk/Red Hat Satellite server.  You access the GUI by typing in the FQDN of the spacewalk server and it will redirect you to the Create Spacewalk Administrator.  You should see a screen much like the following;

Screenshot of first login screen post installation of Spacewalk on CentOS 7

Just enter a few essential details and away you go.  OK you won’t get too far yet, but keep reading!

Upon clicking the “Create Login” button, you should see the normal dashboard screen that is displayed when logging into Spacewalk (and Red Hat Satellite) for the first time.  With one exception.  You should also have a banner across the top with the following wording;

You have created your first user for the Spacewalk Service. Additional configuration should be finalized by Click here

Make sure you click the “Click here” link and make sure you complete the rest of the steps.

General Configuration

I would advise you check and double check the General configuration tab, specifically the Spacewalk hostname, this should ideally match the FQDN of your satellite server.  And if you haven’t specified a name which is resolvable via DNS you will likely find that things don’t run exactly as they should.

The Certificate tab will be of interest if you are minting your own SSL certificates or wish to use a commercially generated cert. The Bootstrap script tab is where you define settings relating to how clients connect and the associated security around those connections.

The Organizations tab (which in my opinion should read Organisations (because that’s how you spell it) is where you can define how your organisation looks, you can define multiple activation keys for different parts of your organisation, manage subscriptions and users.  To name but a few of the things you can do.

Restart tab, er, do I really need to suggest what this does???  And finally the Cobbler tab.  From here you can kick of a synchronisation between Spacewalk and Cobblerd.  I recommend clicking it now to make sure the integration between the two applications is working.  I would also suggest you double check the cobbler log file located at /var/log/cobbler/cobbler.log for any signs of problems.  Here’s a sample output;

Mon Apr 18 23:49:10 2016 - INFO | authenticate; ['toby', True]
Mon Apr 18 23:49:10 2016 - INFO | REMOTE sync; user(toby)
Mon Apr 18 23:49:10 2016 - DEBUG | authorize; ['toby', 'sync', None, None, True]
Mon Apr 18 23:49:10 2016 - DEBUG | REMOTE toby authorization result: True; user(?)
Mon Apr 18 23:49:10 2016 - INFO | sync
Mon Apr 18 23:49:10 2016 - INFO | running pre-sync triggers
Mon Apr 18 23:49:10 2016 - INFO | cleaning trees
Mon Apr 18 23:49:10 2016 - INFO | mkdir: /var/lib/tftpboot/pxelinux.cfg
Mon Apr 18 23:49:10 2016 - INFO | mkdir: /var/lib/tftpboot/grub
Mon Apr 18 23:49:10 2016 - INFO | mkdir: /var/lib/tftpboot/images
Mon Apr 18 23:49:10 2016 - INFO | mkdir: /var/lib/tftpboot/s390x
Mon Apr 18 23:49:10 2016 - INFO | mkdir: /var/lib/tftpboot/ppc
Mon Apr 18 23:49:10 2016 - INFO | mkdir: /var/lib/tftpboot/etc
Mon Apr 18 23:49:10 2016 - INFO | removing: /var/lib/tftpboot/grub/images
Mon Apr 18 23:49:10 2016 - INFO | copying bootloaders
Mon Apr 18 23:49:10 2016 - INFO | copying: /usr/share/syslinux/pxelinux.0 -> /var/lib/tftpboot/pxelinux.0
Mon Apr 18 23:49:10 2016 - INFO | copying: /usr/share/syslinux/menu.c32 -> /var/lib/tftpboot/menu.c32
Mon Apr 18 23:49:10 2016 - INFO | copying: /usr/share/syslinux/memdisk -> /var/lib/tftpboot/memdisk
Mon Apr 18 23:49:10 2016 - INFO | copying distros
Mon Apr 18 23:49:10 2016 - INFO | copying images
Mon Apr 18 23:49:10 2016 - INFO | generating PXE configuration files
Mon Apr 18 23:49:10 2016 - INFO | cleaning link caches
Mon Apr 18 23:49:10 2016 - INFO | generating PXE menu structure
Mon Apr 18 23:49:10 2016 - INFO | running post-sync triggers
Mon Apr 18 23:49:10 2016 - DEBUG | running python triggers from /var/lib/cobbler/triggers/sync/post/*
Mon Apr 18 23:49:10 2016 - DEBUG | running python trigger cobbler.modules.sync_post_restart_services
Mon Apr 18 23:49:10 2016 - DEBUG | running shell triggers from /var/lib/cobbler/triggers/sync/post/*
Mon Apr 18 23:49:10 2016 - DEBUG | running python triggers from /var/lib/cobbler/triggers/change/*
Mon Apr 18 23:49:10 2016 - DEBUG | running python trigger cobbler.modules.scm_track
Mon Apr 18 23:49:10 2016 - DEBUG | running shell triggers from /var/lib/cobbler/triggers/change/*

Generate a default activation key

In order to register systems against your newly installed Spacewalk server, you must have a activation key defined.  This is not done automatically, and therefore we shall tackle it now.  Navigate to Systems > Activation Keys.

Initially you should see a message stating;

You do not currently have a universal default activation key set. To set a key as the universal default, please visit the details page of that key and check off the 'Universal Default?' checkbox.

Click + Create Key in the top right hand corner of the Activation Keys screen.  You will need to add the following details;

  • Description
  • A Key (I would advise putting something meaningful in here, rather than allowing a key to be auto-generated.
  • Leave the Usage field blank
  • Leave the Base Channels as default (Spacewalk Default)
  • Add-on Entitlements, I have selected only Provisioning (it can be changed later)
  • I also ticked the Universal Default as I do not want to restrict its use

After the default key has been created, the screen looks like this;

Image contains example screenshot of Spacewalk activation key.

Creating your first package repository (and channel)

I will be focusing on CentOS 7 here, but Satellite is capable of providing a centralised repository for other RPM based distributions.

CentOS 7 Base Repository

We will assume you may at some point want to build further servers using the base OS RPMs.  The first thing you need to do is find a local mirror site which you can base your repository on. CentOS provide a lovely page on their web site – https://www.centos.org/download/mirrors/, which details, (by country) where you can download the packages from.  In my case I searched the page for United Kingdom and picked one from the list.

Lets get on with the job at hand, and create a repository.  Click Channels > Manage Software Channels > Manage Repositories.  And then click Create Repository.  You will then see a screen, not too dissimilar to the one below;

spacewalk_create_repository

Once you have defined the repository label and URL (this is the source url which Spacewalk will be using to obtain the packages from).  I have defined the SSL cert that was generated during installation.

You will now need to create the Channel that will be associated to this channel.  Click Manage Software Channels from the left hand menu and then click on Create Channel.  You will (once the page has loaded) be given a few ground rules regarding naming conventions and then the opportunity to create your new channel.  The long and short of it is this;

  • Channel Name and Channel Label are required (hence the red asterisk)
  • Channel Name;
    • must be between 6 and 256 characters in length
    • must begin with a letter
    • may contain spaces, parentheses () and forward slashes /.
  • Channel Label must;
    • be no longer than 128 characters
    • start with a letter or digit
  • Must be lowercase (no exceptions)
  • May contain hyphens, periods, underscores and numerals

spacewalk_Create_channel

Some other options on the screen also include controlling access to the repository (i.e. is it private and only accessible to your Spacewalk organisation, or is it public), also you can define GPG security settings for signed packages.

The last step is to marry the repository and channel together.  This is achieved by going to the Repositories tab, and selecting the repository from the list of available repositories.  In my case it is just one.

The last step, will be kick off a synchronisation of the repository.  Now there are two ways to do this; 1) By click the Sync tab, then ticking the “Create kickstartable tree” option and then clicking Sync Now.  Or 2) run the following command from the cli.

/usr/bin/spacewalk-repo-sync --channel centos7base --type yum --latest --sync-kickstart

Now sit back and watch/wait.  For the current 7.2 repo, the base number of packages is just over 9000, so depending on your connection to the internet, you could find this to be a quick process or quite slow (also very dependent upon the mirror you have selected).  Another option which I haven’t don’t but I believe it would work, is to use a copy of the installation media.  If you try that option, let me know how you get on 🙂

Registering your first client

First, you will need to make sure you have the required packages on the client to be register.  In my case, I had used a minimal install and as such I was missing the required packages.  Easily rectified;

[root@rhc-client ~]# yum install rhn-setup
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package rhn-setup.noarch 0:2.0.2-6.el7 will be installed
--> Processing Dependency: rhn-client-tools = 2.0.2-6.el7 for package: rhn-setup-2.0.2-6.el7.noarch
--> Processing Dependency: rhnsd for package: rhn-setup-2.0.2-6.el7.noarch
--> Running transaction check
---> Package rhn-client-tools.noarch 0:2.0.2-6.el7 will be installed
--> Processing Dependency: rhnlib >= 2.5.57 for package: rhn-client-tools-2.0.2-6.el7.noarch
--> Processing Dependency: python-hwdata for package: rhn-client-tools-2.0.2-6.el7.noarch
--> Processing Dependency: python-gudev for package: rhn-client-tools-2.0.2-6.el7.noarch
--> Processing Dependency: python-dmidecode for package: rhn-client-tools-2.0.2-6.el7.noarch
---> Package rhnsd.x86_64 0:5.0.13-5.el7 will be installed
--> Processing Dependency: rhn-check >= 0.0.8 for package: rhnsd-5.0.13-5.el7.x86_64
--> Running transaction check
---> Package python-dmidecode.x86_64 0:3.10.13-11.el7 will be installed
---> Package python-gudev.x86_64 0:147.2-7.el7 will be installed
---> Package python-hwdata.noarch 0:1.7.3-4.el7 will be installed
---> Package rhn-check.noarch 0:2.0.2-6.el7 will be installed
--> Processing Dependency: yum-rhn-plugin >= 1.6.4-1 for package: rhn-check-2.0.2-6.el7.noarch
---> Package rhnlib.noarch 0:2.5.65-2.el7 will be installed
--> Running transaction check
---> Package yum-rhn-plugin.noarch 0:2.0.1-5.el7 will be installed
--> Processing Dependency: m2crypto >= 0.16-6 for package: yum-rhn-plugin-2.0.1-5.el7.noarch
--> Running transaction check
---> Package m2crypto.x86_64 0:0.21.1-17.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch        Version             Repository          Size
================================================================================
Installing:
 rhn-setup             noarch      2.0.2-6.el7         th_lab_server       87 k
Installing for dependencies:
 m2crypto              x86_64      0.21.1-17.el7       th_lab_server      429 k
 python-dmidecode      x86_64      3.10.13-11.el7      th_lab_server       82 k
 python-gudev          x86_64      147.2-7.el7         th_lab_server       18 k
 python-hwdata         noarch      1.7.3-4.el7         th_lab_server       32 k
 rhn-check             noarch      2.0.2-6.el7         th_lab_server       52 k
 rhn-client-tools      noarch      2.0.2-6.el7         th_lab_server      379 k
 rhnlib                noarch      2.5.65-2.el7        th_lab_server       65 k
 rhnsd                 x86_64      5.0.13-5.el7        th_lab_server       48 k
 yum-rhn-plugin        noarch      2.0.1-5.el7         th_lab_server       80 k

Transaction Summary
================================================================================
Install  1 Package (+9 Dependent packages)

Total size: 1.2 M
Total download size: 1.2 M
Installed size: 4.8 M
Is this ok [y/d/N]: y
Downloading packages:
(1/9): python-gudev-147.2-7.el7.x86_64.rpm                 |  18 kB   00:00     
(2/9): m2crypto-0.21.1-17.el7.x86_64.rpm                   | 429 kB   00:00     
(3/9): python-hwdata-1.7.3-4.el7.noarch.rpm                |  32 kB   00:00     
(4/9): rhn-check-2.0.2-6.el7.noarch.rpm                    |  52 kB   00:00     
(5/9): rhn-client-tools-2.0.2-6.el7.noarch.rpm             | 379 kB   00:00     
(6/9): rhn-setup-2.0.2-6.el7.noarch.rpm                    |  87 kB   00:00     
(7/9): rhnlib-2.5.65-2.el7.noarch.rpm                      |  65 kB   00:00     
(8/9): rhnsd-5.0.13-5.el7.x86_64.rpm                       |  48 kB   00:00     
(9/9): yum-rhn-plugin-2.0.1-5.el7.noarch.rpm               |  80 kB   00:00     
--------------------------------------------------------------------------------
Total                                              1.3 MB/s | 1.2 MB  00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python-gudev-147.2-7.el7.x86_64                             1/10 
  Installing : rhnlib-2.5.65-2.el7.noarch                                  2/10 
  Installing : python-hwdata-1.7.3-4.el7.noarch                            3/10 
  Installing : python-dmidecode-3.10.13-11.el7.x86_64                      4/10 
  Installing : rhn-client-tools-2.0.2-6.el7.noarch                         5/10 
  Installing : m2crypto-0.21.1-17.el7.x86_64                               6/10 
  Installing : rhnsd-5.0.13-5.el7.x86_64                                   7/10 
  Installing : rhn-setup-2.0.2-6.el7.noarch                                8/10 
  Installing : yum-rhn-plugin-2.0.1-5.el7.noarch                           9/10 
  Installing : rhn-check-2.0.2-6.el7.noarch                               10/10 
  Verifying  : rhn-setup-2.0.2-6.el7.noarch                                1/10 
  Verifying  : m2crypto-0.21.1-17.el7.x86_64                               2/10 
  Verifying  : rhn-check-2.0.2-6.el7.noarch                                3/10 
  Verifying  : python-dmidecode-3.10.13-11.el7.x86_64                      4/10 
  Verifying  : rhnsd-5.0.13-5.el7.x86_64                                   5/10 
  Verifying  : rhn-client-tools-2.0.2-6.el7.noarch                         6/10 
  Verifying  : python-hwdata-1.7.3-4.el7.noarch                            7/10 
  Verifying  : yum-rhn-plugin-2.0.1-5.el7.noarch                           8/10 
  Verifying  : rhnlib-2.5.65-2.el7.noarch                                  9/10 
  Verifying  : python-gudev-147.2-7.el7.x86_64                            10/10 

Installed:
  rhn-setup.noarch 0:2.0.2-6.el7                                                

Dependency Installed:
  m2crypto.x86_64 0:0.21.1-17.el7      python-dmidecode.x86_64 0:3.10.13-11.el7 
  python-gudev.x86_64 0:147.2-7.el7    python-hwdata.noarch 0:1.7.3-4.el7       
  rhn-check.noarch 0:2.0.2-6.el7       rhn-client-tools.noarch 0:2.0.2-6.el7    
  rhnlib.noarch 0:2.5.65-2.el7         rhnsd.x86_64 0:5.0.13-5.el7              
  yum-rhn-plugin.noarch 0:2.0.1-5.el7 

Complete!

Next step is to install your Spacewalk server’s ssl certificate on the client.  This is a security measure which enables the client to verify that the server it is talking to is really the server it SHOULD be talking too.

[toby@devops ~]$ sudo rpm -Uvh http://manager/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
Retrieving http://manager/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:rhn-org-trusted-ssl-cert-1.0-1   ################################# [100%]

The final step in the process is to actually register the client against the Spacewalk/Satellite server.

[toby@devops ~]$ sudo rhnreg_ks --serverUrl=https://manager.lab.tobyheywood.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=1-lab.tobyheywood.com
This system is not subscribed to any channels.
RHN channel support will be disabled.

At this point, we float over to the Spacewalk server UI, and we should now see our client in the list of Systems;

Screenshot showing the newly register client listed under the Systems list in Spacewalk

Now, for those of you who have a keen eye for detail, you will have noticed in the screenshot above and the snippet of command line output at the time of registration, that the system isn’t currently subscribed to any channels.  This is very easily remedied;

  1. Click on the client in the system list
  2. On the initial Overview screen, you will see a box – Subscribed Channels
  3. Click Alter Channel Subscriptions
  4. Now Select from the list under Base Software Channel – in my case “centos_7_base”
  5. Click confirm
  6. You should now see the channel listed under the heading “Software Channel Subscriptions”.
  7. In addition you may have child channels created beneath your base channel.

And there we have it.  Time to have a play and see what you can do by having a click around the tabs related to the system and the wider Spacewalk UI.

Reference material

https://fedorahosted.org/spacewalk/wiki/RegisteringClients

Featured image credit:  Thanks to NASA for making the image of Tim Peake on his spacewalk free to use!

Warning /dev/root does not exist – The Devil is in the Detail

Posted on Leave a commentPosted in CentOS 7, Deployment, Linux, Red Hat Satelitte Server, RHEL 7, Systems Management

Following on from an earlier post, it would seem that the “Warning /dev/root does not exist” issue is not confined tononekickstart pxe booted installations as I had first thought.

I was working on a RHEL 7 installation using Red Hat Satellite 5.7 (upgrade to 6.x in the pipeline but bigger fish to fry right now), where we were re-using a lot of the RHEL 6 pxelinux kernel parameter’s.

Now as you may or may not know (if you have read my other posts on the topic), there are numerous Anaconda and dracut parameter’s that can be passed to the kernel in the pxelinux.cfg/default (or the mac specific) config file.  The problem we had found was the existence off a ksdevice= parameter which pointed to eth0.  In RHEL/CentOS 7, the ethernet device naming standard changes from ethX to ensX, which works out as follows;

  • en = Ethernet
  • sX = Slot X (where X is the physical or virtual slot number where the nic resides)

By default the first interface is used by anaconda/dracut/pxelinux, IF no option is specified.  If however you specifically tell it to use something which fundamentally doesn’t exist, it WILL still try to use that… and fail!  Miserably!  And give you an error which ultimately seems kind of unrelated.

You have been warned!

As with many things in life, this served as a reminder that things change and that you can’t always take the “old” and reuse with the “new” without issue.

Featured image credit:  Thanks to (Elba) Dave Shewmaker for taking the rather weird picture which resembles The Devil in the Detail and posting it on Flickr.

CentOS 7 – Watch out for nmtui

Posted on 3 CommentsPosted in CentOS 7, Linux, Networking, RHEL 7, System Administration

It would appear that I have been caught out twice now due to the way that nmtui (Network Manager Text User Interface) works.  I have been messing around with various internal sandboxed networks in my VM environment and (I can only assume ion my haste), I have entered the IP address of a second NIC without full regard for the on screen prompts.

In nmtui, there is one field missing which is quite common in many other tools.  Take a look at the following and tell me what’s missing;

nmtui_sandbox_config

So what field do you think is missing?

Now although the information is all on screen in the screenshot above, there is one thing that may not be obvious.  In the Addresses field you specify not only the IP address but also the subnet mask in CIDR (which stands for Classless Inter-Domain Routing) notation.

IF, you happen to enter and IP address without thinking about it and don’t specify the netmask or CIDR, nmtui assumes that you are only referring to a /32, a.k.a. a netmask of 255.255.255.255, which for the uninitiated means just that IP.  If assumes that there is nothing beyond that IP address.  It’s world is only itself.

In the good old days where I used to configure the IP address via ifcfg-eth* files, I also remembered to enter the NETMASK= line, and therefore never had this issue.

Anyway rant over.  Hopefully twice is enough, because if nothing else, if I have name resolutions errors in my logs again, I will be making sure my netmask is set correctly, before thinking that tomcat is having issues.

Grrrrrr

Featured image credit:  Thanks to versageek for making the Network Spagetti image available on Flickr.com.

Spacewalk – Post install sanity check

Posted on 1 CommentPosted in CentOS 7, Deployment, Fedora, Linux, RHEL 7, Spacewalk, System Administration, Systems Management

After having installed Spacewalk, got it working to a certain point and then found that there may have been issues with the installation, I thought it would be easier to simply re-install spacewalk onto a new virtual machine.

So following on from my how to article, I wanted to make sure that post installation, I had performed sufficient checks to confirm that there were no issues with the scheduler service or cobbler, as these were two things I had great difficulty trying to get working.

I guess it is also worth mentioning that the VM I am running spacewalk on has a single vCPU and 4GB of memory.  For storage I have given it 40G which will do me fine.  And as for the OS it is running CentOS 7 (1511).

So what should we check

Good question.  The following is a rough list of all the services I confirmed as enabled, running and also that there were no horrible errors in the log files

Services

  • cobblerd
  • postgresql
  • xinetd (tftp)
  • httpd
  • tomcat
  • taskomatic (a.k.a. the scheduler)

cobblerd

[toby@manager ~]$ sudo systemctl status cobblerd
● cobblerd.service - LSB: daemon for libvirt virtualization API
   Loaded: loaded (/etc/rc.d/init.d/cobblerd)
   Active: active (running) since Fri 2016-04-15 23:08:37 BST; 24min ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/cobblerd.service
           └─13257 /usr/bin/python -s /bin/cobblerd --daemonize

Apr 15 23:08:35 manager systemd[1]: Starting LSB: daemon for libvirt virtualization API...
Apr 15 23:08:37 manager cobblerd[13247]: Starting cobbler daemon: [  OK  ]
Apr 15 23:08:37 manager systemd[1]: Started LSB: daemon for libvirt virtualization API.
Apr 15 23:31:35 manager systemd[1]: [/run/systemd/generator.late/cobblerd.service:8] Failed to add dependency on network,.service, ignoring: Invalid argument
Apr 15 23:31:35 manager systemd[1]: [/run/systemd/generator.late/cobblerd.service:8] Failed to add dependency on xinetd,.service, ignoring: Invalid argument

The last two lines can be ignore.  I believe this is purely due some references to the sysvinit scripts which are no longer used, and as you will see later things appear to be running fine (this time around)

PostgreSQL

[toby@manager ~]$ sudo systemctl status postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-15 23:05:39 BST; 35min ago
 Main PID: 12556 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─12556 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─12557 postgres: logger process   
           ├─12559 postgres: checkpointer process   
           ├─12560 postgres: writer process   
           ├─12561 postgres: wal writer process   
           ├─12562 postgres: autovacuum launcher process   
           ├─12563 postgres: stats collector process   
           ├─13191 postgres: rhnuser rhnschema [local] idle in transaction
           ├─13343 postgres: rhnuser rhnschema 127.0.0.1(55225) idle
           ├─13344 postgres: rhnuser rhnschema 127.0.0.1(55226) idle
           ├─13345 postgres: rhnuser rhnschema 127.0.0.1(55227) idle
           ├─13346 postgres: rhnuser rhnschema 127.0.0.1(55228) idle
           ├─13347 postgres: rhnuser rhnschema 127.0.0.1(55229) idle
           ├─13348 postgres: rhnuser rhnschema 127.0.0.1(55230) idle
           ├─13350 postgres: rhnuser rhnschema 127.0.0.1(55231) idle
           ├─13351 postgres: rhnuser rhnschema 127.0.0.1(55232) idle
           ├─13352 postgres: rhnuser rhnschema 127.0.0.1(55233) idle
           ├─13354 postgres: rhnuser rhnschema 127.0.0.1(55234) idle
           ├─13355 postgres: rhnuser rhnschema 127.0.0.1(55235) idle
           ├─13356 postgres: rhnuser rhnschema 127.0.0.1(55236) idle
           ├─13357 postgres: rhnuser rhnschema 127.0.0.1(55237) idle
           ├─13358 postgres: rhnuser rhnschema 127.0.0.1(55238) idle
           ├─13361 postgres: rhnuser rhnschema 127.0.0.1(55240) idle
           ├─13391 postgres: rhnuser rhnschema 127.0.0.1(55244) idle
           ├─13442 postgres: rhnuser rhnschema 127.0.0.1(55246) idle
           ├─13444 postgres: rhnuser rhnschema 127.0.0.1(55248) idle
           ├─13451 postgres: rhnuser rhnschema 127.0.0.1(55250) idle
           ├─13651 postgres: rhnuser rhnschema 127.0.0.1(55266) idle
           ├─28774 postgres: rhnuser rhnschema 127.0.0.1(55272) idle
           ├─28842 postgres: rhnuser rhnschema 127.0.0.1(55275) idle
           ├─28843 postgres: rhnuser rhnschema 127.0.0.1(55276) idle
           ├─28844 postgres: rhnuser rhnschema 127.0.0.1(55277) idle
           ├─28847 postgres: rhnuser rhnschema 127.0.0.1(55278) idle
           ├─28902 postgres: rhnuser rhnschema 127.0.0.1(55279) idle
           └─28903 postgres: rhnuser rhnschema 127.0.0.1(55280) idle

Apr 15 23:05:38 manager systemd[1]: Starting PostgreSQL database server...
Apr 15 23:05:39 manager systemd[1]: Started PostgreSQL database server.

tftp (by way of xinetd)

[toby@manager ~]$ sudo systemctl enable tftp
[toby@manager ~]$ sudo systemctl start tftp
[toby@manager ~]$ sudo systemctl status tftp
● tftp.service - Tftp Server
   Loaded: loaded (/usr/lib/systemd/system/tftp.service; indirect; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-15 23:46:30 BST; 2s ago
     Docs: man:in.tftpd
 Main PID: 29012 (in.tftpd)
   CGroup: /system.slice/tftp.service
           └─29012 /usr/sbin/in.tftpd -s /var/lib/tftpboot

httpd

[toby@manager ~]$ sudo systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-15 23:08:34 BST; 39min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 13168 (httpd)
   Status: "Total requests: 1; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─13168 /usr/sbin/httpd -DFOREGROUND
           ├─13169 /usr/sbin/httpd -DFOREGROUND
           ├─13170 /usr/sbin/httpd -DFOREGROUND
           ├─13171 /usr/sbin/httpd -DFOREGROUND
           ├─13172 /usr/sbin/httpd -DFOREGROUND
           ├─13173 /usr/sbin/httpd -DFOREGROUND
           ├─13174 /usr/sbin/httpd -DFOREGROUND
           ├─13175 /usr/sbin/httpd -DFOREGROUND
           └─13176 /usr/sbin/httpd -DFOREGROUND

Apr 15 23:08:34 manager systemd[1]: Starting The Apache HTTP Server...
Apr 15 23:08:34 manager httpd[13168]: AH00557: httpd: apr_sockaddr_info_get() failed for manager
Apr 15 23:08:34 manager httpd[13168]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
Apr 15 23:08:34 manager systemd[1]: Started The Apache HTTP Server.

Feel free to ignore the warning messages with regards to the FQDN.

tomcat

Once the install had completed there was an error message;

Tomcat failed to start properly or the installer ran out of tries. Please check /var/log/tomcat*/catalina.out for errors.

I checked the logs and saw some errors, but as you can see from the following, simple making sure it was enabled and started appears to have cleared up what ever the issue may have been.

[toby@manager ~]$ sudo systemctl enable tomcat
[toby@manager ~]$ sudo systemctl start tomcat
[toby@manager ~]$ sudo systemctl status tomcat
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-15 23:05:39 BST; 43min ago
 Main PID: 12589 (java)
   CGroup: /system.slice/tomcat.service
           └─12589 /usr/lib/jvm/jre/bin/java -ea -Xms256m -Xmx256m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024 -XX...

Apr 15 23:08:33 manager server[12589]: INFO: Deployment of configuration descriptor /etc/tomcat/Catalina/localhost/rhn.xml has finished in 172,857 ms
Apr 15 23:08:33 manager server[12589]: Apr 15, 2016 11:08:33 PM org.apache.coyote.AbstractProtocol start
Apr 15 23:08:33 manager server[12589]: INFO: Starting ProtocolHandler ["http-bio-127.0.0.1-8080"]
Apr 15 23:08:34 manager server[12589]: Apr 15, 2016 11:08:34 PM org.apache.coyote.AbstractProtocol start
Apr 15 23:08:34 manager server[12589]: INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"]
Apr 15 23:08:34 manager server[12589]: Apr 15, 2016 11:08:34 PM org.apache.coyote.AbstractProtocol start
Apr 15 23:08:34 manager server[12589]: INFO: Starting ProtocolHandler ["ajp-bio-0:0:0:0:0:0:0:1-8009"]
Apr 15 23:08:34 manager server[12589]: Apr 15, 2016 11:08:34 PM org.apache.catalina.startup.Catalina start
Apr 15 23:08:34 manager server[12589]: INFO: Server startup in 173112 ms
Apr 15 23:31:41 manager systemd[1]: Started Apache Tomcat Web Application Container.

taskomatic

Now, this one doesn’t appear to have been moved over to the new systemd environment and therefore we resort back to the good old sysvinit scripts and the service command to confirm this one is working;

[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (13296).

Looking good so far

But can it withstand a reboot?  Now that is the question.  So I repeated the above steps again, just to confirm.  I won’t bore you with all the details;

  • cobblerd.service – active (running)
  • httpd.service – active (running)
  • tftp.service – inactive (dead)
  • postgresql.service – active (running)
  • tomcat.service –active (running)
  • taskomatic – RHN Taskomatic is not running.

taskomatic revisited

[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is not running.
[toby@manager ~]$ sudo chkconfig taskomatic on
[toby@manager ~]$ sudo service taskomatic start
Starting RHN Taskomatic...
[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (10870).

And for good measure I gave the machine another reboot, just to confirm the taskomatic service did start.

[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (1278).

Oh Yeah!  Now I’m a happy camper.  And it’s time to re-visit the initial configuration part, which I shall post about shortly.

Image credit; Thanks to Mark Walsh for making the featured image called “Russell Street Court Cells – Padded Cell” available on Flickr.com.

DKIM + Courier-MTA on CentOS7

Posted on 8 CommentsPosted in CentOS 7, DNS, Email, Linux, RHEL 7, Security, SELinux, System Administration

Email.  One of those things which is critical to business and depending on what you read and who you speak to, you may be led to believe that it is dying out in favour of instant messaging technologies.  Well, as of right now, I can’t see it dying out any time soon, though it does come with a really annoying, frustrating, excruciating problem.  Junk mail!  Otherwise know as spam!  It has once again become a bit of a problem for me, especially when the likes of yahoo begin to imped the flow of emails leaving one of the servers I have worked on recently unable to send emails to recipients with Yahoo accounts.

Now, don’t get my wrong.  I don’t hold a grudge with any of the large email service providers for blocking spam which has sadly managed to be sent via a mail server I have some involvement with, but I do wish the world wasn’t such a bad place that we have to continuously fight on our email frontiers to protect our virtual/real name and brand.

So anyway.  On with the show

The server

  • CentOS 7
  • Patched to the nines
  • Courier MTA
  • Blocking of known spammers was enabled
  • Spamassassin was used as a secondary measure should the hard blocks fail
  • SPF is configured for all hosted domains
  • DKIM was not configured
  • DMAC was not configured
  • It *was not* an open relay
  • The time was not in sync, which  didn’t help later down the line

The problem

The server’s “reputation” had been tarnished as someone had managed to start sending spam via the server through a vulnerability which was found in one of the websites hosted on the server.

After reviewing the advice from Yahoo, I started my investigation on how to approach this topic.

A bit of googling, led me to this web site; https://www.strangeworld.com/blog/archives/128. Now for me, there were a few things missing in the instructions, though complete were it was most needed.  So recreating some of the  good work done on the strangeworld.com website and providing my own twist on things here;

DKIM & ZDKIMFILTER Pre-requisites

In order to install zdkimfilter you need to manually compile the code and install (a .spec file will be included in future releases which should mean you can just use rpmbuild to create the RPM package for you).  The machine I needed to install this onto did not have any compilers install for reasons of security, and so my list of pre-requisites was rather long and included more than I would have liked but needs must, and the compilers were removed afterwards.  But anyway, here is the list of commands issued to get everything where it needed to be;

sudo rpm -ivh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
sudo yum install gcc
sudo yum install openssl-libs openssl-devel
sudo yum install libopendkim opendkim
sudo yum install libtool libidn2 libidn2-devel
sudo yum install opendbx-devel opendbx-mysql opendbx-utils opendbx

Note.  The addition of the elrepo.org rpm is required as there are a couple of dependencies, which don’t appear to be available in the main CentOS yum repository.

Download, compile and install

Now, the original instructions I followed, (see reference material at bottom of this page), took me through the steps of manually compiling the code for and installing opendkim.  After I had installed it, I found that opendkim is available in the CentOS base yum repo and so I could have avoided that step.  You will be pleased to hear that I have not included those steps here (if you do need to know though, see this blog post – strangeworld.com).

Next up we need to obtain the tar ball containing the source code for zdkimfilter, which was downloaded from; http://www.tana.it/sw/zdkimfilter/.  At the time of writing the current version was v1.5.

Now we have that lets work through the process of compiling and installing it.

wget http://www.tana.it/sw/zdkimfilter/zdkimfilter-1.5.tar.gz
tar zxvf zdkimfilter-1.5.tar.gz 
cd zdkimfilter-1.5
./configure --prefix=/usr/local
make -j4
sudo make install

Configuration

Last on the list of installation steps is configuring zdkimfilter and making sure we have the correct directory structure, etc;

cp /etc/courier/filters/zdkimfilter.conf.dist /etc/courier/filters/zdkimfilter.conf
mkdir /etc/courier/filters/keys
chmod u=rwx,g=rw,o-rwx /etc/courier/filters/keys

Nothing more was needed at this time.

Generate some keys and update your DNS zones

At this point, we should have everything installed and configured that we need, and can now proceed with the steps for generating the necessary private keys which will be used to sign our emails as they leave your courier email server and also the public key that will be published in DNS.

A word of warning.  Not sure how important this is, but when looking at generating the keys using opendkim-genkey, I would advise that you choose your selector name carefully.  Initially I set this to be the same as the domain.  I then went to a couple of dkim testing websites and found that one of them didn’t like the “.” (dot) in the selector name.  At this point I changed my plan slightly (in case this was a more wide spread issue) and made sure there were no periods in the selector name.

cd /etc/courier/filters/keys/
opendkim-genkey -b 2048 -d tobyheywood.com -D /etc/courier/filters/keys -s tobyheywood -r --nosubdomains -v
ln -s tobyheywood.private tobyheywood.com
chown root.daemon tobyheywood.*
(Just making sure ownership is correct)
chmod u=rw,g=r,o-rwx tobyheywood.*
(this will change the permissions on the tobyheywood.private file which contains the private key, and the tobyheywood.txt file which contains the public key)

It’s also worth making sure the directory /etc/courier/filters/keys has the right permissions;

chmod u=rwx,g=rx,o-rwx /etc/courier/filters/keys

And also if you are running SELinux, then it is worth double checking the SELinux security context associated with these newly created files.  In my case, I didn’t have to do anything, but best to check.

Now you need to update your DNS zone file(s) with everything in the .txt file up to and including the last “)”.  You could include the rest if you wish, but it’s purely a comment.  Reload you zone and test.

Testing, testing, 1, 2, 3

I performed two types of test.  The first was to make sure the record was returned correctly when queried by third parties and that there were no issues with the DNS side of things.  The website I preferred was; https://www.mail-tester.com/spf-dkim-check as it was a nice and simple site and worked better than the unnamed first site I tried.

The second part of my test, was to send an email to a yahoo or gmail account to confirm that it was accepted and so that I could review the headers.  This turned out to be a very good move!  The headers of my first test emails showed there was no DKIM key present, which I thought was odd, but that was due to the umask and the keys directory didn’t have the execute bit set for group members.  I have corrected this in the above [rough] instructions.

Another attempt at sending an email showed there was a valid key but that the time on this server was ahead by some 6 minutes.  I then found that NTP hadn’t been enabled and so had to enable this in order to get the time aligned with other servers on the Internet.

My last test email, was a complete success.  The headers showed that everything was as it should be.  The DKIM signature had been accepted.

The next thing I need to read up on and implement for the domain in question is dmarc.  But more on that later.

Reference material

Courier MTA and DKIM

Image credit:  Thanks to Jake Rush for uploading the featured image GotCredit to flickr.com.

Who is this Microsoft and what have you done with the REAL Microsoft?!?!

Posted on Leave a commentPosted in Uncategorized

So, it would appear that Microsoft have turned another corner, with regards to opening up it’s platform to a wider audience!  They will be bringing the bash shell to Windows 10 this summer!

See this tech crunch article on this very topic!

http://techcrunch.com/2016/03/30/be-very-afraid-hell-has-frozen-over-bash-is-coming-to-windows-10/?ncid=tcdaily

I’m quite excited about it really, as it will mean that I can work from a Windows 10 machine without having to constantly go back and forth between dev machine (windows) and dev server.

What’s next Microsoft?

Back to basics – Kickstart your anaconda file (your systems blueprint)

Posted on Leave a commentPosted in CentOS 7, Deployment, Linux, RHEL 7, System Administration

The final piece in this jigsaw puzzle of network installation madness is the kickstart file.  It is the blueprint from which your RHEL/CentOS/Fedora/[enter distribution name here] is built from.

My preferred way of creating the initial template is to perform a manual installation and then to tweak the resultant /root/anaconda-ks.cfg file to my needs.  This is also the recommended method in the Red Hat documentation.  By doing it this way you can avoid some of the pitfalls of parameter changes within the kickstart script which may alter between releases of RHEL/CentOS.

So, here is what mine currently looks like;

[root@rhc-server ~]# cat anaconda-ks.cfg 
#version=RHEL7
# System authorization information
auth --enableshadow --passalgo=sha512

# Use CDROM installation media
cdrom
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='gb'
# System language
lang en_GB.UTF-8

# Network information
network  --bootproto=dhcp --device=ens3 --ipv6=auto --activate
network  --hostname=rhc-server
# Root password
rootpw --iscrypted $6$1RcgxNlQgKVJAEHj$dcQtZ3Jhe8vw1Aj3rB8zyLl2tTumL88czrcJo4nyUeashp7/rJvE3lFOmWsu9Ml.AZw7PSx5u1M0IGeTLa5ds.
# System timezone
timezone Europe/London --isUtc
user --groups=wheel --name=toby --password=$6$wRgpQD/lrXaqF8cW$PdCIE3CdBq5PxWYejCSpWFVuiMGUIs.eKQXPR5pDeHTmKp3/10qazLDVaCJcpp7zenDKWNqWPXrYjEGRJsAK41 --iscrypted --gecos="toby"
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
autopart --type=lvm
# Partition clearing information
clearpart --none --initlabel 

%packages
@core

%end

Over time, the kickstart file will evolve, and it is a wise man or woman who validates the configuration before actually trying to use the file.  But, having said that, the ksvalidator utility (as the documentation states), can only validate things so far and it will not look at the %pre, %post or %packages sections.  It also doesn’t guarantee a successful install, it just provides a sanity check before you really test it.

So anyway, here is my slightly modified version of the kickstart file, which I have saved in a directory which is accessible via the web server.

[root@rhc-server ks]# cat /var/www/html/ks/basic.ks 
# System authorization information
auth --enableshadow --passalgo=sha512

# Use installation files via http
install
url --url=http://rhc-server.lab.tobyheywood.com/centos7/

# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda

# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='gb'

# System language
lang en_GB.UTF-8

# Network information
network  --bootproto=dhcp --device=ens3 --ipv6=auto --activate

# Root password
rootpw --iscrypted $6$1RcgxNlQgKVJAEHj$dcQtZ3Jhe8vw1Aj3rB8zyLl2tTumL88czrcJo4nyUeashp7/rJvE3lFOmWsu9Ml.AZw7PSx5u1M0IGeTLa5ds.

# System timezone
timezone Europe/London --isUtc
user --groups=wheel --name=toby --password=$6$wRgpQD/lrXaqF8cW$PdCIE3CdBq5PxWYejCSpWFVuiMGUIs.eKQXPR5pDeHTmKp3/10qazLDVaCJcpp7zenDKWNqWPXrYjEGRJsAK41 --iscrypted --gecos="toby"

# System bootloader configuration
bootloader --location=mbr --boot-drive=vda  --iscrypted --password=grub.pbkdf2.sha512.10000.AD234D09B3DDE933C60C46AAA52B95DBB2D48D766E8DDD1852FBF373C8D0474401E4FD6D1D997B1D169B35C69D7776B3FCAC6A3BB6338B0910EB0899B0452BFE.531AE5C20F8CA36DB321770537C0C872B4670EB60F9461A85D2D429C36BAC80F12EBDCC85A6514889332B70BBA780285F84DFDCA57A3B92C3F9FA7387F9F59A0

autopart --type=lvm
# Partition clearing information
clearpart --none --initlabel 

# Firewall & SELinux settings
firewall --enabled --ssh
selinux --enforcing

# Create yum .repo file
repo --name=TheLab --baseurl=http://rhc-server.lab.tobyheywood.com/centos7/ --install

%packages
@core

%end

reboot

I have highlighted all lines that I have modified and will explain any I have added along the way.

So what did I change?

  • Line 6 & 7 – These two lines are required to specify where the install files are actually located.  Note this does have to be the same structure as the ISO (just in case you were wondering).
  • Line 11 – Just in case we have any other disks available to us, we focus the installation to use only the first virtIO disk it sees.  Another point to make here is that device naming is not guaranteed and the documentation dues make some alternative suggestions here.
  • Line 30 – I updated the boot-drive parameter so that it correctly reflected the hard drive device naming and also followed a best practice of implementing a password for the grub2 bootloader.
  • Line 37 & 38 – I explicitly enable selinux in enforcing mode and also set a very basic firewall rule
  • Line 41 – Rather than having to go in and modify my own yum .repo file you can use this option with the install flag to set this up for you

Not many changes but hopefully it provides a good idea of what is needed to get a working install fully operational.

Validating your kickstart script

As mentioned earlier, it is worth validating you kickstart script simply as a sanity check, this can be done as follows;

Installing the ksvalidtor app

[root@rhc-server ks]# yum install pykickstart
Loaded plugins: fastestmirror
baselocal                                                                                                            | 3.6 kB  00:00:00     
(1/2): baselocal/group_gz                                                                                            | 155 kB  00:00:00     
(2/2): baselocal/primary_db                                                                                          | 2.8 MB  00:00:00     
Determining fastest mirrors
Resolving Dependencies
--> Running transaction check
---> Package pykickstart.noarch 0:1.99.66.6-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                           Arch                         Version                               Repository                       Size
============================================================================================================================================
Installing:
 pykickstart                       noarch                       1.99.66.6-1.el7                       baselocal                       328 k

Transaction Summary
============================================================================================================================================
Install  1 Package

Total download size: 328 k
Installed size: 1.5 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : pykickstart-1.99.66.6-1.el7.noarch                                                                                       1/1 
  Verifying  : pykickstart-1.99.66.6-1.el7.noarch                                                                                       1/1 

Installed:
  pykickstart.noarch 0:1.99.66.6-1.el7                                                                                                      

Complete!

Running ksvalidator

Note.  If there is no output after running the command it is a good sign!  Just for good measure I also like to echo out the return code following the command just to give me that extra warm feeling :).

[root@rhc-server ks]# ksvalidator /var/www/html/ks/basic.ks 
[root@rhc-server ks]# echo $?
0

The last thing we need to do is make sure this will be used, or as a bare minimum, that it can be used.  So lets edit the pxelinux.cfg/default file;

[root@rhc-server ks]# cat /tftpboot/pxelinux.cfg/default 
DEFAULT menu.c32
PROMPT 0
TIMEOUT 300
ONTIMEOUT localdisk
MENU TITLE PXE Network Boot

LABEL localdisk
    MENU LABEL ^Local Hard Drive
    MENU DEFAULT
    LOCALBOOT 0

LABEL Install_CentOS_7_2
    MENU LABEL CentOS 7.2
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/images/pxeboot/initrd.img inst.repo=http://rhc-server.lab.tobyheywood.com/centos7 inst.geoloc=0

LABEL Install_CentOS_7_2_KS
    MENU LABEL CentOS 7.2 (KS)
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/images/pxeboot/initrd.img inst.ks=http://rhc-server.lab.tobyheywood.com/ks/basic.ks inst.geoloc=0

The key thing here is that I have done away with the “repo.inst” parameter and it’s associated value and replaced it with the  “inst.ks” parameter, which points to the kickstart file I created earlier.

And that is it, we have a very basic automated installation of CentOS/RHEL 7 over the network without having to pick up a single ISO image and burn it to disk, or USB stick, or manually mount it to a VM.

Reference Material

Credit to Will Scullin, who made his Blueprint image available on Flickr.com.

Back to basics – Setting up a TFTP server & PXE (PreBoot Environment)

Posted on Leave a commentPosted in CentOS 7, DHCP, DNS, Fedora, RHCE, RHEL 6, RHEL 7, System Administration

Right then.  So far (if you have been following along) we have done the following;

  • Created a local yum repository based on the installation media on the initial server in our sand boxed lab
  • Installed and setup DNS with Forward and Reverse zones
  • Installed and configured dhcpd for the lab network
  • Tested the DNS and dhcp services using a client machine
  • And network enabled our yum repository by exposing the directory using Apache

The final steps to enable an ISO free installation of CentOS 7 into a KVM virtual machine are;

  • Installing, configuring and testing Trivial FTP, adding additional configuration to DHCPd to enable PXE booting and testing that setup (this post)
  • The final piece will be to create our kickstart file from which will define the standard installation on CentOS 7 (maybe splitting out into server and client)

So, lets not waste any time and get our hands dirty and flex our fingers with a bit of typing…

Trivial FTP (tftp)

First things first, lets log on to the server and get the packages installed.  Thankfully they are part of the installation media and therefore part of the yum repository that was set up in the last post.

[toby@rhc-server ~]$ sudo yum install tftp*
Loaded plugins: fastestmirror
baselocal                                                                                                            | 3.6 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
-- Running transaction check
--- Package tftp.x86_64 0:5.2-11.el7 will be installed
--- Package tftp-server.x86_64 0:5.2-11.el7 will be installed
-- Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                            Arch                          Version                            Repository                        Size
============================================================================================================================================
Installing:
 tftp                               x86_64                        5.2-11.el7                         baselocal                         35 k
 tftp-server                        x86_64                        5.2-11.el7                         baselocal                         44 k

Transaction Summary
============================================================================================================================================
Install  2 Packages

Total download size: 79 k
Installed size: 112 k
Is this ok [y/d/N]: y
Downloading packages:
--------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                       654 kB/s |  79 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : tftp-server-5.2-11.el7.x86_64                                                                                            1/2 
  Installing : tftp-5.2-11.el7.x86_64                                                                                                   2/2 
  Verifying  : tftp-5.2-11.el7.x86_64                                                                                                   1/2 
  Verifying  : tftp-server-5.2-11.el7.x86_64                                                                                            2/2 

Installed:
  tftp.x86_64 0:5.2-11.el7                                          tftp-server.x86_64 0:5.2-11.el7                                         

Complete!

Note, I’ve installed both client and server RPMs on the server. For my tests from a client I will only install the client tftp package.

Next step is to prepare the folder structure where the files will be served from.  Don’t forget to make sure SELinux contexts, etc. are defined correctly otherwise things will not work as expected.

[root@rhc-server lib]# ll -aZ /var/lib/tftpboot
drwxr-xr-x. root root system_u:object_r:tftpdir_rw_t:s0 .
drwxr-xr-x. root root system_u:object_r:var_lib_t:s0    ..

Note that the SELinux type is “tftpdir_rw_t”, we will need to apply this same type to the folder created next.

[root@rhc-server ~]# mkdir /tftpboot
[root@rhc-server ~]# chcon --reference=/var/lib/tftpboot/ /tftpboot
[root@rhc-server ~]# ll -Z /
[.. snip ..]
drwxr-xr-x. root root system_u:object_r:tftpdir_rw_t:s0 tftpboot

I am slightly cheating with the above command as I am using the standard tftp directory SELinux security types and contexts as a reference. Why make things more difficult than they need to be.

And now lets configure the tftp daemon to use this new location by default.

So lets just check where we are;

  • tftp server and client installed? – check!
  • folder structure created, permissions set and SELinux attributes defined correctly – check!
  • tftp server configured? – Nope

Best fix the TFTP configuration side of things before going further;

[toby@rhc-server lib]$ cat /etc/xinetd.d/tftp
# default: off
# description: The tftp server serves files using the trivial file transfer \
#    protocol.  The tftp protocol is often used to boot diskless \
#    workstations, download configuration files to network-aware printers, \
#    and to start the installation process for some operating systems.
service tftp
{
socket_type        = dgram
protocol        = udp
wait            = yes
user            = root
server            = /usr/sbin/in.tftpd
server_args        = -s /tftpboot
disable            = no
per_source        = 11
cps            = 100 2
flags            = IPv4
}

OK, so the remaining task at this point before we start the service is to make sure the firewall is configured to allow connectivity to the tftp service via its LAN interface and also to make sure the hosts.allow file has an entry for the tftp service.  hosts.allow is used by the xinetd processes, and is required in addition to the firewall changes.

Lets get the hosts.allow file out of the way first;

[root@rhc-server ~]# cat /etc/hosts.allow 
#
# hosts.allow	This file contains access rules which are used to
#		allow or deny connections to network services that
#		either use the tcp_wrappers library or that have been
#		started through a tcp_wrappers-enabled xinetd.
#
#		See 'man 5 hosts_options' and 'man 5 hosts_access'
#		for information on rule syntax.
#		See 'man tcpd' for information on tcp_wrappers
#
in.tftp:	ALL

And now for the firewall;

[root@rhc-server ~]# firewall-cmd --zone public --add-service tftp
[root@rhc-server ~]# firewall-cmd --permanent --zone public --add-service tftp
success

It is worth mentioning that if you use the “–permanent” parameter on the command line, it will not be applied immediately.  Now we should be good to start the service and do some tests. We will create a test file to try and copy via tftp before performing the tests.

[root@rhc-server ~]# systemctl enable tftp.socket
ln -s '/usr/lib/systemd/system/tftp.socket' '/etc/systemd/system/sockets.target.wants/tftp.socket'
[root@rhc-server ~]# systemctl reload xinetd

Based on the above the service has started successfully and nothing appears to be out of the ordinary in the journal, so lets proceed with the testing, and here is my test file;

[root@rhc-server ~]# echo "Hello?  Is it me you're looking for?" > /tftpboot/this_is_a_test

[root@rhc-server ~]# ll -Z /tftpboot/
-rw-r--r--. root root unconfined_u:object_r:tftpdir_rw_t:s0 this_is_a_test

Call me paranoid, but I wanted to make sure the file had inherited the SELinux type of “tftpdir_rw_t”.  Which it did 🙂

Testing locally on server

[toby@rhc-server ~]$ tftp -4 localhost
tftp> get this_is_a_test
tftp> quit
[toby@rhc-server ~]$ ls
this_is_a_test
[toby@rhc-server ~]$ cat this_is_a_test 
Hello?  Is it me you're looking for?

Looking good so far! 🙂

Testing remotely from client

Before we can test lets determine if the tftp package is installed, in my case it wasn’t, so I installed it;

[toby@rhc-client ~]$ yum search tftp
Loaded plugins: fastestmirror, langpacks
Determining fastest mirrors
============================================================ N/S matched: tftp =============================================================
syslinux-tftpboot.x86_64 : SYSLINUX modules in /tftpboot, available for network booting
tftp.x86_64 : The client for the Trivial File Transfer Protocol (TFTP)
tftp-server.x86_64 : The server for the Trivial File Transfer Protocol (TFTP)

  Name and summary matches only, use "search all" for everything.
[toby@rhc-client ~]$ yum list tftp
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Available Packages
tftp.x86_64                                                     5.2-11.el7                                                     th_lab_server
[toby@rhc-client ~]$ sudo yum install tftp
[sudo] password for toby: 
Loaded plugins: fastestmirror, langpacks
th_lab_server                                                                                                        | 3.6 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package tftp.x86_64 0:5.2-11.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                      Arch                           Version                            Repository                             Size
============================================================================================================================================
Installing:
 tftp                         x86_64                         5.2-11.el7                         th_lab_server                          35 k

Transaction Summary
============================================================================================================================================
Install  1 Package

Total download size: 35 k
Installed size: 48 k
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/th_lab_server/packages/tftp-5.2-11.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for tftp-5.2-11.el7.x86_64.rpm is not installed
tftp-5.2-11.el7.x86_64.rpm                                                                                           |  35 kB  00:00:00     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-0.1406.el7.centos.2.3.x86_64 (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : tftp-5.2-11.el7.x86_64                                                                                                   1/1 
  Verifying  : tftp-5.2-11.el7.x86_64                                                                                                   1/1 

Installed:
  tftp.x86_64 0:5.2-11.el7                                                                                                                  

Complete!

And now the test from the client.

[toby@rhc-client ~]$ sudo firewall-cmd --zone public --add-service tftp
[sudo] password for toby: 
success
[toby@rhc-client ~]$ tftp rhc-server
tftp> get this_is_a_test
tftp> quit
[toby@rhc-client ~]$ ls
Desktop  Documents  Downloads  Music  Pictures  Public  Templates  test  this_is_a_test  Videos
[toby@rhc-client ~]$ cat this_is_a_test 
Hello?  Is it me you're looking for?
[toby@rhc-client ~]$ sudo firewall-cmd --zone public --remove-service tftp
success

Note. If you don’t temporarily enable the tftp port on the client, the test will fail. I got the follow error via tcpdump which highlighted that the firewall was blocking the request;

23:25:40.388650 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 54)
    192.168.20.50.36588 > 192.168.20.1.69: [udp sum ok]  26 RRQ "this_is_a_test" netascii
23:25:40.390312 IP (tos 0x0, ttl 64, id 31747, offset 0, flags [none], proto UDP (17), length 70)
    192.168.20.1.43788 > 192.168.20.50.36588: [bad udp cksum 0xa9c7 -> 0xddd1!] UDP, length 42
23:25:40.390721 IP (tos 0xc0, ttl 64, id 3881, offset 0, flags [none], proto ICMP (1), length 98)
    192.168.20.50 > 192.168.20.1: ICMP host 192.168.20.50 unreachable - admin prohibited, length 78
	IP (tos 0x0, ttl 64, id 31747, offset 0, flags [none], proto UDP (17), length 70)
    192.168.20.1.43788 > 192.168.20.50.36588: [udp sum ok] UDP, length 42

At this point we have now completed the necessary steps to ensure we have a working tftp service.

Setting up PXE

The second part of today’s post covers the steps needed to enable booting from the network to facilitate building machines without the hassle of creating USB bootable images or burning ISO images to CD or DVD.

Out task list for this section is;

  • Add the additional config to the DHCP scope
  • Ensure we have the pxelinux/syslinux installed and copied to the required location
  • Create a basic menu to provide end users with a installation options
  • Test

Configuring DHCP

So, lets start by reminding ourselves what the current dhcpd.conf file looks like;

[toby@rhc-server ~]$ sudo cat /etc/dhcp/dhcpd.conf
[sudo] password for toby: 
#
#  lab.tobyheywood.com dhcp daemon configuration file
#
#  2016-02-22 - Initial creation
#

# Define which IP to listen on.  NOTE. daemon can only listen to one
# IP at a time if defined.
local-address 192.168.20.1;

# option definitions common to all supported networks...
option domain-name "lab.tobyheywood.com";
option domain-name-servers ns.lab.tobyheywood.com;

default-lease-time 600;
max-lease-time 7200;

# Use this to enble / disable dynamic dns updates globally.
#ddns-update-style interim;

# This is the authoritative DHCP server.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# Interface which can be accessed from outside the sandbox
# **** NOT IN USE ****
subnet 192.168.122.0 netmask 255.255.255.0 {
}

# The lab network
subnet 192.168.20.0 netmask 255.255.255.128 {
  range 192.168.20.50 192.168.20.99;
  option routers rtr.lab.tobyheywood.com;
}

So in order to get PXE (Preboot eXecution Environment) to function we will need to add a few more lines to the configuration, as highlighted below.

[toby@rhc-server ~]$ sudo cat /etc/dhcp/dhcpd.conf
[sudo] password for toby: 
#
#  lab.tobyheywood.com dhcp daemon configuration file
#
#  2016-02-22 - Initial creation
#

# Define which IP to listen on.  NOTE. daemon can only listen to one
# IP at a time if defined.
local-address 192.168.20.1;

# option definitions common to all supported networks...
option domain-name "lab.tobyheywood.com";
option domain-name-servers ns.lab.tobyheywood.com;

default-lease-time 600;
max-lease-time 7200;

# Use this to enble / disable dynamic dns updates globally.
#ddns-update-style interim;

# This is the authoritative DHCP server.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# Interface which can be accessed from outside the sandbox
# **** NOT IN USE ****
subnet 192.168.122.0 netmask 255.255.255.0 {
}

# The lab network
subnet 192.168.20.0 netmask 255.255.255.128 {
  range 192.168.20.50 192.168.20.99;
  option routers rtr.lab.tobyheywood.com;
}

# Additional configuration for PXE booting
allow booting;
allow bootp;
option option-128 code 128 = string;
option option-129 code 129 = text;
next-server 192.168.20.1;
filename "/pxelinux.0";

Setting up the required syslinux files in /tftpboot

During the section above when testing the tftp service on the client, I saw that there may be a shortcut to getting things up and running.  Everything I have read says you need to manually copy the syslinux files into the /tftpboot directory, however, if you search the rpm database with yum for anything tftp related, I see that there is potentially an easier way.

[toby@rhc-server ~]$ yum list tftp
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Installed Packages
tftp.x86_64                                     5.2-11.el7                                     @baselocal
[toby@rhc-server ~]$ yum search tftp
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
=========================================== N/S matched: tftp ===========================================
syslinux-tftpboot.x86_64 : SYSLINUX modules in /tftpboot, available for network booting
tftp.x86_64 : The client for the Trivial File Transfer Protocol (TFTP)
tftp-server.x86_64 : The server for the Trivial File Transfer Protocol (TFTP)

As you can see there appears to be a syslinux-tftpboot rpm.  So lets see what it gives us;

[toby@rhc-server ~]$ sudo yum install syslinux-tftpboot
[sudo] password for toby: 
Loaded plugins: fastestmirror
baselocal                                                                         | 3.6 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package syslinux-tftpboot.x86_64 0:4.05-8.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================
 Package                        Arch                Version                 Repository              Size
=========================================================================================================
Installing:
 syslinux-tftpboot              x86_64              4.05-8.el7              baselocal              425 k

Transaction Summary
=========================================================================================================
Install  1 Package

Total download size: 425 k
Installed size: 1.3 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : syslinux-tftpboot-4.05-8.el7.x86_64                                                   1/1 
  Verifying  : syslinux-tftpboot-4.05-8.el7.x86_64                                                   1/1 

Installed:
  syslinux-tftpboot.x86_64 0:4.05-8.el7                                                                  

Complete!
[toby@rhc-server ~]$ ls /tftpboot/
cat.c32        dmitest.c32   host.c32     ls.c32       pcitest.c32   rosh.c32        vesamenu.c32
chain.c32      elf.c32       ifcpu64.c32  lua.c32      pmload.c32    sanboot.c32     vpdtest.c32
cmd.c32        ethersel.c32  ifcpu.c32    mboot.c32    poweroff.com  sdi.c32         whichsys.c32
config.c32     gfxboot.c32   ifplop.c32   memdisk      pwd.c32       sysdump.c32     zzjson.c32
cpuid.c32      gpxecmd.c32   int18.com    memdump.com  pxechain.com  this_is_a_test
cpuidtest.c32  gpxelinux.0   kbdmap.c32   meminfo.c32  pxelinux.0    ver.com
disk.c32       hdt.c32       linux.c32    menu.c32     reboot.c32    vesainfo.c32

It looks like we have a winner!  Now we just need to make sure that the Preboot environment has a kernel or two to play with;

[root@rhc-server ~]# cp /software/centos7/images/pxeboot/* /tftpboot/centos7/
[root@rhc-server ~]# ls -Z /tftpboot/centos7/
-rw-r--r--. root root unconfined_u:object_r:tftpdir_t:s0 initrd.img
-r--r--r--. root root unconfined_u:object_r:tftpdir_t:s0 TRANS.TBL
-rw-r--r--. root root unconfined_u:object_r:tftpdir_t:s0 upgrade.img
-rwxr-xr-x. root root unconfined_u:object_r:tftpdir_t:s0 vmlinuz

OK, so lets now create the directory which will hold the PXE menus;

[toby@rhc-server ~]$ sudo mkdir /tftpboot/pxelinux.cfg
[sudo] password for toby: 
[toby@rhc-server ~]$ ls -Zd /tftpboot/pxelinux.cfg
drwxr-xr-x. root root unconfined_u:object_r:tftpdir_t:s0 /tftpboot/pxelinux.cfg

And now lets look at what is required to create the menus from which we will be able to select installation options.

[root@rhc-server centos7]# cat /tftpboot/pxelinux.cfg/default 
DEFAULT menu.c32
PROMPT 0
TIMEOUT 300
ONTIMEOUT localdisk
MENU TITLE PXE Network Boot

LABEL localdisk
    MENU LABEL ^Local Hard Drive
    MENU DEFAULT
    LOCALBOOT 0

LABEL Install_CentOS_7_2
    MENU LABEL CentOS 7.2
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/isolinux/initrd.img  inst.repo=http://rhc-server.lab.tobyheywood.com/centos7

At this point, all I wanted to do was prove that the pxe settings in dhcp were correct and that maybe, just maybe, I could build a vm across the network, first time.  No such luck! 🙁  Well, I guess 50% of the way there.

You can skip the next and jump to here if you don’t want to understand the pain I went through to get this operational.

I thought things were going well.  I created a blank VM, made sure to select that I was going to boot from the network to install the OS in Virtual Manager and turned the VM on.

pxeboot_initial_screen

So far so good!

pxeboot_menu

I’m now feel pretty happy with myself.  I select CentOS 7.0 and hit the enter key… and then am presented with the expected Welcome to CentOS 7 screen, from where I can kick off a manual installation.

pxeboot_centos7_initial_install_screen

So all, in all, things look good.  However there is more work to be done, as the next step is to create a kickstart file to define how the base install should look.

Reference Material

Trivial FTP (tftp)

PXE

Featured image taken by Ed Robinson who kindly uploaded it to Flickr.com.  I have made no changes to this image and left it in its original form for your viewing pleasure.  And to give the page a bit of colour 😉