Spacewalk – Post install sanity check

Posted on 1 CommentPosted in CentOS 7, Deployment, Fedora, Linux, RHEL 7, Spacewalk, System Administration, Systems Management

After having installed Spacewalk, got it working to a certain point and then found that there may have been issues with the installation, I thought it would be easier to simply re-install spacewalk onto a new virtual machine.

So following on from my how to article, I wanted to make sure that post installation, I had performed sufficient checks to confirm that there were no issues with the scheduler service or cobbler, as these were two things I had great difficulty trying to get working.

I guess it is also worth mentioning that the VM I am running spacewalk on has a single vCPU and 4GB of memory.  For storage I have given it 40G which will do me fine.  And as for the OS it is running CentOS 7 (1511).

So what should we check

Good question.  The following is a rough list of all the services I confirmed as enabled, running and also that there were no horrible errors in the log files

Services

  • cobblerd
  • postgresql
  • xinetd (tftp)
  • httpd
  • tomcat
  • taskomatic (a.k.a. the scheduler)

cobblerd

[toby@manager ~]$ sudo systemctl status cobblerd
● cobblerd.service - LSB: daemon for libvirt virtualization API
   Loaded: loaded (/etc/rc.d/init.d/cobblerd)
   Active: active (running) since Fri 2016-04-15 23:08:37 BST; 24min ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/cobblerd.service
           └─13257 /usr/bin/python -s /bin/cobblerd --daemonize

Apr 15 23:08:35 manager systemd[1]: Starting LSB: daemon for libvirt virtualization API...
Apr 15 23:08:37 manager cobblerd[13247]: Starting cobbler daemon: [  OK  ]
Apr 15 23:08:37 manager systemd[1]: Started LSB: daemon for libvirt virtualization API.
Apr 15 23:31:35 manager systemd[1]: [/run/systemd/generator.late/cobblerd.service:8] Failed to add dependency on network,.service, ignoring: Invalid argument
Apr 15 23:31:35 manager systemd[1]: [/run/systemd/generator.late/cobblerd.service:8] Failed to add dependency on xinetd,.service, ignoring: Invalid argument

The last two lines can be ignore.  I believe this is purely due some references to the sysvinit scripts which are no longer used, and as you will see later things appear to be running fine (this time around)

PostgreSQL

[toby@manager ~]$ sudo systemctl status postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-15 23:05:39 BST; 35min ago
 Main PID: 12556 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─12556 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─12557 postgres: logger process   
           ├─12559 postgres: checkpointer process   
           ├─12560 postgres: writer process   
           ├─12561 postgres: wal writer process   
           ├─12562 postgres: autovacuum launcher process   
           ├─12563 postgres: stats collector process   
           ├─13191 postgres: rhnuser rhnschema [local] idle in transaction
           ├─13343 postgres: rhnuser rhnschema 127.0.0.1(55225) idle
           ├─13344 postgres: rhnuser rhnschema 127.0.0.1(55226) idle
           ├─13345 postgres: rhnuser rhnschema 127.0.0.1(55227) idle
           ├─13346 postgres: rhnuser rhnschema 127.0.0.1(55228) idle
           ├─13347 postgres: rhnuser rhnschema 127.0.0.1(55229) idle
           ├─13348 postgres: rhnuser rhnschema 127.0.0.1(55230) idle
           ├─13350 postgres: rhnuser rhnschema 127.0.0.1(55231) idle
           ├─13351 postgres: rhnuser rhnschema 127.0.0.1(55232) idle
           ├─13352 postgres: rhnuser rhnschema 127.0.0.1(55233) idle
           ├─13354 postgres: rhnuser rhnschema 127.0.0.1(55234) idle
           ├─13355 postgres: rhnuser rhnschema 127.0.0.1(55235) idle
           ├─13356 postgres: rhnuser rhnschema 127.0.0.1(55236) idle
           ├─13357 postgres: rhnuser rhnschema 127.0.0.1(55237) idle
           ├─13358 postgres: rhnuser rhnschema 127.0.0.1(55238) idle
           ├─13361 postgres: rhnuser rhnschema 127.0.0.1(55240) idle
           ├─13391 postgres: rhnuser rhnschema 127.0.0.1(55244) idle
           ├─13442 postgres: rhnuser rhnschema 127.0.0.1(55246) idle
           ├─13444 postgres: rhnuser rhnschema 127.0.0.1(55248) idle
           ├─13451 postgres: rhnuser rhnschema 127.0.0.1(55250) idle
           ├─13651 postgres: rhnuser rhnschema 127.0.0.1(55266) idle
           ├─28774 postgres: rhnuser rhnschema 127.0.0.1(55272) idle
           ├─28842 postgres: rhnuser rhnschema 127.0.0.1(55275) idle
           ├─28843 postgres: rhnuser rhnschema 127.0.0.1(55276) idle
           ├─28844 postgres: rhnuser rhnschema 127.0.0.1(55277) idle
           ├─28847 postgres: rhnuser rhnschema 127.0.0.1(55278) idle
           ├─28902 postgres: rhnuser rhnschema 127.0.0.1(55279) idle
           └─28903 postgres: rhnuser rhnschema 127.0.0.1(55280) idle

Apr 15 23:05:38 manager systemd[1]: Starting PostgreSQL database server...
Apr 15 23:05:39 manager systemd[1]: Started PostgreSQL database server.

tftp (by way of xinetd)

[toby@manager ~]$ sudo systemctl enable tftp
[toby@manager ~]$ sudo systemctl start tftp
[toby@manager ~]$ sudo systemctl status tftp
● tftp.service - Tftp Server
   Loaded: loaded (/usr/lib/systemd/system/tftp.service; indirect; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-15 23:46:30 BST; 2s ago
     Docs: man:in.tftpd
 Main PID: 29012 (in.tftpd)
   CGroup: /system.slice/tftp.service
           └─29012 /usr/sbin/in.tftpd -s /var/lib/tftpboot

httpd

[toby@manager ~]$ sudo systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-15 23:08:34 BST; 39min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 13168 (httpd)
   Status: "Total requests: 1; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─13168 /usr/sbin/httpd -DFOREGROUND
           ├─13169 /usr/sbin/httpd -DFOREGROUND
           ├─13170 /usr/sbin/httpd -DFOREGROUND
           ├─13171 /usr/sbin/httpd -DFOREGROUND
           ├─13172 /usr/sbin/httpd -DFOREGROUND
           ├─13173 /usr/sbin/httpd -DFOREGROUND
           ├─13174 /usr/sbin/httpd -DFOREGROUND
           ├─13175 /usr/sbin/httpd -DFOREGROUND
           └─13176 /usr/sbin/httpd -DFOREGROUND

Apr 15 23:08:34 manager systemd[1]: Starting The Apache HTTP Server...
Apr 15 23:08:34 manager httpd[13168]: AH00557: httpd: apr_sockaddr_info_get() failed for manager
Apr 15 23:08:34 manager httpd[13168]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
Apr 15 23:08:34 manager systemd[1]: Started The Apache HTTP Server.

Feel free to ignore the warning messages with regards to the FQDN.

tomcat

Once the install had completed there was an error message;

Tomcat failed to start properly or the installer ran out of tries. Please check /var/log/tomcat*/catalina.out for errors.

I checked the logs and saw some errors, but as you can see from the following, simple making sure it was enabled and started appears to have cleared up what ever the issue may have been.

[toby@manager ~]$ sudo systemctl enable tomcat
[toby@manager ~]$ sudo systemctl start tomcat
[toby@manager ~]$ sudo systemctl status tomcat
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-04-15 23:05:39 BST; 43min ago
 Main PID: 12589 (java)
   CGroup: /system.slice/tomcat.service
           └─12589 /usr/lib/jvm/jre/bin/java -ea -Xms256m -Xmx256m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024 -XX...

Apr 15 23:08:33 manager server[12589]: INFO: Deployment of configuration descriptor /etc/tomcat/Catalina/localhost/rhn.xml has finished in 172,857 ms
Apr 15 23:08:33 manager server[12589]: Apr 15, 2016 11:08:33 PM org.apache.coyote.AbstractProtocol start
Apr 15 23:08:33 manager server[12589]: INFO: Starting ProtocolHandler ["http-bio-127.0.0.1-8080"]
Apr 15 23:08:34 manager server[12589]: Apr 15, 2016 11:08:34 PM org.apache.coyote.AbstractProtocol start
Apr 15 23:08:34 manager server[12589]: INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"]
Apr 15 23:08:34 manager server[12589]: Apr 15, 2016 11:08:34 PM org.apache.coyote.AbstractProtocol start
Apr 15 23:08:34 manager server[12589]: INFO: Starting ProtocolHandler ["ajp-bio-0:0:0:0:0:0:0:1-8009"]
Apr 15 23:08:34 manager server[12589]: Apr 15, 2016 11:08:34 PM org.apache.catalina.startup.Catalina start
Apr 15 23:08:34 manager server[12589]: INFO: Server startup in 173112 ms
Apr 15 23:31:41 manager systemd[1]: Started Apache Tomcat Web Application Container.

taskomatic

Now, this one doesn’t appear to have been moved over to the new systemd environment and therefore we resort back to the good old sysvinit scripts and the service command to confirm this one is working;

[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (13296).

Looking good so far

But can it withstand a reboot?  Now that is the question.  So I repeated the above steps again, just to confirm.  I won’t bore you with all the details;

  • cobblerd.service – active (running)
  • httpd.service – active (running)
  • tftp.service – inactive (dead)
  • postgresql.service – active (running)
  • tomcat.service –active (running)
  • taskomatic – RHN Taskomatic is not running.

taskomatic revisited

[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is not running.
[toby@manager ~]$ sudo chkconfig taskomatic on
[toby@manager ~]$ sudo service taskomatic start
Starting RHN Taskomatic...
[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (10870).

And for good measure I gave the machine another reboot, just to confirm the taskomatic service did start.

[toby@manager ~]$ sudo service taskomatic status
RHN Taskomatic is running (1278).

Oh Yeah!  Now I’m a happy camper.  And it’s time to re-visit the initial configuration part, which I shall post about shortly.

Image credit; Thanks to Mark Walsh for making the featured image called “Russell Street Court Cells – Padded Cell” available on Flickr.com.

DKIM + Courier-MTA on CentOS7

Posted on 8 CommentsPosted in CentOS 7, DNS, Email, Linux, RHEL 7, Security, SELinux, System Administration

Email.  One of those things which is critical to business and depending on what you read and who you speak to, you may be led to believe that it is dying out in favour of instant messaging technologies.  Well, as of right now, I can’t see it dying out any time soon, though it does come with a really annoying, frustrating, excruciating problem.  Junk mail!  Otherwise know as spam!  It has once again become a bit of a problem for me, especially when the likes of yahoo begin to imped the flow of emails leaving one of the servers I have worked on recently unable to send emails to recipients with Yahoo accounts.

Now, don’t get my wrong.  I don’t hold a grudge with any of the large email service providers for blocking spam which has sadly managed to be sent via a mail server I have some involvement with, but I do wish the world wasn’t such a bad place that we have to continuously fight on our email frontiers to protect our virtual/real name and brand.

So anyway.  On with the show

The server

  • CentOS 7
  • Patched to the nines
  • Courier MTA
  • Blocking of known spammers was enabled
  • Spamassassin was used as a secondary measure should the hard blocks fail
  • SPF is configured for all hosted domains
  • DKIM was not configured
  • DMAC was not configured
  • It *was not* an open relay
  • The time was not in sync, which  didn’t help later down the line

The problem

The server’s “reputation” had been tarnished as someone had managed to start sending spam via the server through a vulnerability which was found in one of the websites hosted on the server.

After reviewing the advice from Yahoo, I started my investigation on how to approach this topic.

A bit of googling, led me to this web site; https://www.strangeworld.com/blog/archives/128. Now for me, there were a few things missing in the instructions, though complete were it was most needed.  So recreating some of the  good work done on the strangeworld.com website and providing my own twist on things here;

DKIM & ZDKIMFILTER Pre-requisites

In order to install zdkimfilter you need to manually compile the code and install (a .spec file will be included in future releases which should mean you can just use rpmbuild to create the RPM package for you).  The machine I needed to install this onto did not have any compilers install for reasons of security, and so my list of pre-requisites was rather long and included more than I would have liked but needs must, and the compilers were removed afterwards.  But anyway, here is the list of commands issued to get everything where it needed to be;

sudo rpm -ivh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
sudo yum install gcc
sudo yum install openssl-libs openssl-devel
sudo yum install libopendkim opendkim
sudo yum install libtool libidn2 libidn2-devel
sudo yum install opendbx-devel opendbx-mysql opendbx-utils opendbx

Note.  The addition of the elrepo.org rpm is required as there are a couple of dependencies, which don’t appear to be available in the main CentOS yum repository.

Download, compile and install

Now, the original instructions I followed, (see reference material at bottom of this page), took me through the steps of manually compiling the code for and installing opendkim.  After I had installed it, I found that opendkim is available in the CentOS base yum repo and so I could have avoided that step.  You will be pleased to hear that I have not included those steps here (if you do need to know though, see this blog post – strangeworld.com).

Next up we need to obtain the tar ball containing the source code for zdkimfilter, which was downloaded from; http://www.tana.it/sw/zdkimfilter/.  At the time of writing the current version was v1.5.

Now we have that lets work through the process of compiling and installing it.

wget http://www.tana.it/sw/zdkimfilter/zdkimfilter-1.5.tar.gz
tar zxvf zdkimfilter-1.5.tar.gz 
cd zdkimfilter-1.5
./configure --prefix=/usr/local
make -j4
sudo make install

Configuration

Last on the list of installation steps is configuring zdkimfilter and making sure we have the correct directory structure, etc;

cp /etc/courier/filters/zdkimfilter.conf.dist /etc/courier/filters/zdkimfilter.conf
mkdir /etc/courier/filters/keys
chmod u=rwx,g=rw,o-rwx /etc/courier/filters/keys

Nothing more was needed at this time.

Generate some keys and update your DNS zones

At this point, we should have everything installed and configured that we need, and can now proceed with the steps for generating the necessary private keys which will be used to sign our emails as they leave your courier email server and also the public key that will be published in DNS.

A word of warning.  Not sure how important this is, but when looking at generating the keys using opendkim-genkey, I would advise that you choose your selector name carefully.  Initially I set this to be the same as the domain.  I then went to a couple of dkim testing websites and found that one of them didn’t like the “.” (dot) in the selector name.  At this point I changed my plan slightly (in case this was a more wide spread issue) and made sure there were no periods in the selector name.

cd /etc/courier/filters/keys/
opendkim-genkey -b 2048 -d tobyheywood.com -D /etc/courier/filters/keys -s tobyheywood -r --nosubdomains -v
ln -s tobyheywood.private tobyheywood.com
chown root.daemon tobyheywood.*
(Just making sure ownership is correct)
chmod u=rw,g=r,o-rwx tobyheywood.*
(this will change the permissions on the tobyheywood.private file which contains the private key, and the tobyheywood.txt file which contains the public key)

It’s also worth making sure the directory /etc/courier/filters/keys has the right permissions;

chmod u=rwx,g=rx,o-rwx /etc/courier/filters/keys

And also if you are running SELinux, then it is worth double checking the SELinux security context associated with these newly created files.  In my case, I didn’t have to do anything, but best to check.

Now you need to update your DNS zone file(s) with everything in the .txt file up to and including the last “)”.  You could include the rest if you wish, but it’s purely a comment.  Reload you zone and test.

Testing, testing, 1, 2, 3

I performed two types of test.  The first was to make sure the record was returned correctly when queried by third parties and that there were no issues with the DNS side of things.  The website I preferred was; https://www.mail-tester.com/spf-dkim-check as it was a nice and simple site and worked better than the unnamed first site I tried.

The second part of my test, was to send an email to a yahoo or gmail account to confirm that it was accepted and so that I could review the headers.  This turned out to be a very good move!  The headers of my first test emails showed there was no DKIM key present, which I thought was odd, but that was due to the umask and the keys directory didn’t have the execute bit set for group members.  I have corrected this in the above [rough] instructions.

Another attempt at sending an email showed there was a valid key but that the time on this server was ahead by some 6 minutes.  I then found that NTP hadn’t been enabled and so had to enable this in order to get the time aligned with other servers on the Internet.

My last test email, was a complete success.  The headers showed that everything was as it should be.  The DKIM signature had been accepted.

The next thing I need to read up on and implement for the domain in question is dmarc.  But more on that later.

Reference material

Courier MTA and DKIM

Image credit:  Thanks to Jake Rush for uploading the featured image GotCredit to flickr.com.

Back to basics – Kickstart your anaconda file (your systems blueprint)

Posted on Leave a commentPosted in CentOS 7, Deployment, Linux, RHEL 7, System Administration

The final piece in this jigsaw puzzle of network installation madness is the kickstart file.  It is the blueprint from which your RHEL/CentOS/Fedora/[enter distribution name here] is built from.

My preferred way of creating the initial template is to perform a manual installation and then to tweak the resultant /root/anaconda-ks.cfg file to my needs.  This is also the recommended method in the Red Hat documentation.  By doing it this way you can avoid some of the pitfalls of parameter changes within the kickstart script which may alter between releases of RHEL/CentOS.

So, here is what mine currently looks like;

[root@rhc-server ~]# cat anaconda-ks.cfg 
#version=RHEL7
# System authorization information
auth --enableshadow --passalgo=sha512

# Use CDROM installation media
cdrom
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='gb'
# System language
lang en_GB.UTF-8

# Network information
network  --bootproto=dhcp --device=ens3 --ipv6=auto --activate
network  --hostname=rhc-server
# Root password
rootpw --iscrypted $6$1RcgxNlQgKVJAEHj$dcQtZ3Jhe8vw1Aj3rB8zyLl2tTumL88czrcJo4nyUeashp7/rJvE3lFOmWsu9Ml.AZw7PSx5u1M0IGeTLa5ds.
# System timezone
timezone Europe/London --isUtc
user --groups=wheel --name=toby --password=$6$wRgpQD/lrXaqF8cW$PdCIE3CdBq5PxWYejCSpWFVuiMGUIs.eKQXPR5pDeHTmKp3/10qazLDVaCJcpp7zenDKWNqWPXrYjEGRJsAK41 --iscrypted --gecos="toby"
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
autopart --type=lvm
# Partition clearing information
clearpart --none --initlabel 

%packages
@core

%end

Over time, the kickstart file will evolve, and it is a wise man or woman who validates the configuration before actually trying to use the file.  But, having said that, the ksvalidator utility (as the documentation states), can only validate things so far and it will not look at the %pre, %post or %packages sections.  It also doesn’t guarantee a successful install, it just provides a sanity check before you really test it.

So anyway, here is my slightly modified version of the kickstart file, which I have saved in a directory which is accessible via the web server.

[root@rhc-server ks]# cat /var/www/html/ks/basic.ks 
# System authorization information
auth --enableshadow --passalgo=sha512

# Use installation files via http
install
url --url=http://rhc-server.lab.tobyheywood.com/centos7/

# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda

# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='gb'

# System language
lang en_GB.UTF-8

# Network information
network  --bootproto=dhcp --device=ens3 --ipv6=auto --activate

# Root password
rootpw --iscrypted $6$1RcgxNlQgKVJAEHj$dcQtZ3Jhe8vw1Aj3rB8zyLl2tTumL88czrcJo4nyUeashp7/rJvE3lFOmWsu9Ml.AZw7PSx5u1M0IGeTLa5ds.

# System timezone
timezone Europe/London --isUtc
user --groups=wheel --name=toby --password=$6$wRgpQD/lrXaqF8cW$PdCIE3CdBq5PxWYejCSpWFVuiMGUIs.eKQXPR5pDeHTmKp3/10qazLDVaCJcpp7zenDKWNqWPXrYjEGRJsAK41 --iscrypted --gecos="toby"

# System bootloader configuration
bootloader --location=mbr --boot-drive=vda  --iscrypted --password=grub.pbkdf2.sha512.10000.AD234D09B3DDE933C60C46AAA52B95DBB2D48D766E8DDD1852FBF373C8D0474401E4FD6D1D997B1D169B35C69D7776B3FCAC6A3BB6338B0910EB0899B0452BFE.531AE5C20F8CA36DB321770537C0C872B4670EB60F9461A85D2D429C36BAC80F12EBDCC85A6514889332B70BBA780285F84DFDCA57A3B92C3F9FA7387F9F59A0

autopart --type=lvm
# Partition clearing information
clearpart --none --initlabel 

# Firewall & SELinux settings
firewall --enabled --ssh
selinux --enforcing

# Create yum .repo file
repo --name=TheLab --baseurl=http://rhc-server.lab.tobyheywood.com/centos7/ --install

%packages
@core

%end

reboot

I have highlighted all lines that I have modified and will explain any I have added along the way.

So what did I change?

  • Line 6 & 7 – These two lines are required to specify where the install files are actually located.  Note this does have to be the same structure as the ISO (just in case you were wondering).
  • Line 11 – Just in case we have any other disks available to us, we focus the installation to use only the first virtIO disk it sees.  Another point to make here is that device naming is not guaranteed and the documentation dues make some alternative suggestions here.
  • Line 30 – I updated the boot-drive parameter so that it correctly reflected the hard drive device naming and also followed a best practice of implementing a password for the grub2 bootloader.
  • Line 37 & 38 – I explicitly enable selinux in enforcing mode and also set a very basic firewall rule
  • Line 41 – Rather than having to go in and modify my own yum .repo file you can use this option with the install flag to set this up for you

Not many changes but hopefully it provides a good idea of what is needed to get a working install fully operational.

Validating your kickstart script

As mentioned earlier, it is worth validating you kickstart script simply as a sanity check, this can be done as follows;

Installing the ksvalidtor app

[root@rhc-server ks]# yum install pykickstart
Loaded plugins: fastestmirror
baselocal                                                                                                            | 3.6 kB  00:00:00     
(1/2): baselocal/group_gz                                                                                            | 155 kB  00:00:00     
(2/2): baselocal/primary_db                                                                                          | 2.8 MB  00:00:00     
Determining fastest mirrors
Resolving Dependencies
--> Running transaction check
---> Package pykickstart.noarch 0:1.99.66.6-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                           Arch                         Version                               Repository                       Size
============================================================================================================================================
Installing:
 pykickstart                       noarch                       1.99.66.6-1.el7                       baselocal                       328 k

Transaction Summary
============================================================================================================================================
Install  1 Package

Total download size: 328 k
Installed size: 1.5 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : pykickstart-1.99.66.6-1.el7.noarch                                                                                       1/1 
  Verifying  : pykickstart-1.99.66.6-1.el7.noarch                                                                                       1/1 

Installed:
  pykickstart.noarch 0:1.99.66.6-1.el7                                                                                                      

Complete!

Running ksvalidator

Note.  If there is no output after running the command it is a good sign!  Just for good measure I also like to echo out the return code following the command just to give me that extra warm feeling :).

[root@rhc-server ks]# ksvalidator /var/www/html/ks/basic.ks 
[root@rhc-server ks]# echo $?
0

The last thing we need to do is make sure this will be used, or as a bare minimum, that it can be used.  So lets edit the pxelinux.cfg/default file;

[root@rhc-server ks]# cat /tftpboot/pxelinux.cfg/default 
DEFAULT menu.c32
PROMPT 0
TIMEOUT 300
ONTIMEOUT localdisk
MENU TITLE PXE Network Boot

LABEL localdisk
    MENU LABEL ^Local Hard Drive
    MENU DEFAULT
    LOCALBOOT 0

LABEL Install_CentOS_7_2
    MENU LABEL CentOS 7.2
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/images/pxeboot/initrd.img inst.repo=http://rhc-server.lab.tobyheywood.com/centos7 inst.geoloc=0

LABEL Install_CentOS_7_2_KS
    MENU LABEL CentOS 7.2 (KS)
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/images/pxeboot/initrd.img inst.ks=http://rhc-server.lab.tobyheywood.com/ks/basic.ks inst.geoloc=0

The key thing here is that I have done away with the “repo.inst” parameter and it’s associated value and replaced it with the  “inst.ks” parameter, which points to the kickstart file I created earlier.

And that is it, we have a very basic automated installation of CentOS/RHEL 7 over the network without having to pick up a single ISO image and burn it to disk, or USB stick, or manually mount it to a VM.

Reference Material

Credit to Will Scullin, who made his Blueprint image available on Flickr.com.

Back to basics – Setting up a TFTP server & PXE (PreBoot Environment)

Posted on Leave a commentPosted in CentOS 7, DHCP, DNS, Fedora, RHCE, RHEL 6, RHEL 7, System Administration

Right then.  So far (if you have been following along) we have done the following;

  • Created a local yum repository based on the installation media on the initial server in our sand boxed lab
  • Installed and setup DNS with Forward and Reverse zones
  • Installed and configured dhcpd for the lab network
  • Tested the DNS and dhcp services using a client machine
  • And network enabled our yum repository by exposing the directory using Apache

The final steps to enable an ISO free installation of CentOS 7 into a KVM virtual machine are;

  • Installing, configuring and testing Trivial FTP, adding additional configuration to DHCPd to enable PXE booting and testing that setup (this post)
  • The final piece will be to create our kickstart file from which will define the standard installation on CentOS 7 (maybe splitting out into server and client)

So, lets not waste any time and get our hands dirty and flex our fingers with a bit of typing…

Trivial FTP (tftp)

First things first, lets log on to the server and get the packages installed.  Thankfully they are part of the installation media and therefore part of the yum repository that was set up in the last post.

[toby@rhc-server ~]$ sudo yum install tftp*
Loaded plugins: fastestmirror
baselocal                                                                                                            | 3.6 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
-- Running transaction check
--- Package tftp.x86_64 0:5.2-11.el7 will be installed
--- Package tftp-server.x86_64 0:5.2-11.el7 will be installed
-- Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                            Arch                          Version                            Repository                        Size
============================================================================================================================================
Installing:
 tftp                               x86_64                        5.2-11.el7                         baselocal                         35 k
 tftp-server                        x86_64                        5.2-11.el7                         baselocal                         44 k

Transaction Summary
============================================================================================================================================
Install  2 Packages

Total download size: 79 k
Installed size: 112 k
Is this ok [y/d/N]: y
Downloading packages:
--------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                       654 kB/s |  79 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : tftp-server-5.2-11.el7.x86_64                                                                                            1/2 
  Installing : tftp-5.2-11.el7.x86_64                                                                                                   2/2 
  Verifying  : tftp-5.2-11.el7.x86_64                                                                                                   1/2 
  Verifying  : tftp-server-5.2-11.el7.x86_64                                                                                            2/2 

Installed:
  tftp.x86_64 0:5.2-11.el7                                          tftp-server.x86_64 0:5.2-11.el7                                         

Complete!

Note, I’ve installed both client and server RPMs on the server. For my tests from a client I will only install the client tftp package.

Next step is to prepare the folder structure where the files will be served from.  Don’t forget to make sure SELinux contexts, etc. are defined correctly otherwise things will not work as expected.

[root@rhc-server lib]# ll -aZ /var/lib/tftpboot
drwxr-xr-x. root root system_u:object_r:tftpdir_rw_t:s0 .
drwxr-xr-x. root root system_u:object_r:var_lib_t:s0    ..

Note that the SELinux type is “tftpdir_rw_t”, we will need to apply this same type to the folder created next.

[root@rhc-server ~]# mkdir /tftpboot
[root@rhc-server ~]# chcon --reference=/var/lib/tftpboot/ /tftpboot
[root@rhc-server ~]# ll -Z /
[.. snip ..]
drwxr-xr-x. root root system_u:object_r:tftpdir_rw_t:s0 tftpboot

I am slightly cheating with the above command as I am using the standard tftp directory SELinux security types and contexts as a reference. Why make things more difficult than they need to be.

And now lets configure the tftp daemon to use this new location by default.

So lets just check where we are;

  • tftp server and client installed? – check!
  • folder structure created, permissions set and SELinux attributes defined correctly – check!
  • tftp server configured? – Nope

Best fix the TFTP configuration side of things before going further;

[toby@rhc-server lib]$ cat /etc/xinetd.d/tftp
# default: off
# description: The tftp server serves files using the trivial file transfer \
#    protocol.  The tftp protocol is often used to boot diskless \
#    workstations, download configuration files to network-aware printers, \
#    and to start the installation process for some operating systems.
service tftp
{
socket_type        = dgram
protocol        = udp
wait            = yes
user            = root
server            = /usr/sbin/in.tftpd
server_args        = -s /tftpboot
disable            = no
per_source        = 11
cps            = 100 2
flags            = IPv4
}

OK, so the remaining task at this point before we start the service is to make sure the firewall is configured to allow connectivity to the tftp service via its LAN interface and also to make sure the hosts.allow file has an entry for the tftp service.  hosts.allow is used by the xinetd processes, and is required in addition to the firewall changes.

Lets get the hosts.allow file out of the way first;

[root@rhc-server ~]# cat /etc/hosts.allow 
#
# hosts.allow	This file contains access rules which are used to
#		allow or deny connections to network services that
#		either use the tcp_wrappers library or that have been
#		started through a tcp_wrappers-enabled xinetd.
#
#		See 'man 5 hosts_options' and 'man 5 hosts_access'
#		for information on rule syntax.
#		See 'man tcpd' for information on tcp_wrappers
#
in.tftp:	ALL

And now for the firewall;

[root@rhc-server ~]# firewall-cmd --zone public --add-service tftp
[root@rhc-server ~]# firewall-cmd --permanent --zone public --add-service tftp
success

It is worth mentioning that if you use the “–permanent” parameter on the command line, it will not be applied immediately.  Now we should be good to start the service and do some tests. We will create a test file to try and copy via tftp before performing the tests.

[root@rhc-server ~]# systemctl enable tftp.socket
ln -s '/usr/lib/systemd/system/tftp.socket' '/etc/systemd/system/sockets.target.wants/tftp.socket'
[root@rhc-server ~]# systemctl reload xinetd

Based on the above the service has started successfully and nothing appears to be out of the ordinary in the journal, so lets proceed with the testing, and here is my test file;

[root@rhc-server ~]# echo "Hello?  Is it me you're looking for?" > /tftpboot/this_is_a_test

[root@rhc-server ~]# ll -Z /tftpboot/
-rw-r--r--. root root unconfined_u:object_r:tftpdir_rw_t:s0 this_is_a_test

Call me paranoid, but I wanted to make sure the file had inherited the SELinux type of “tftpdir_rw_t”.  Which it did 🙂

Testing locally on server

[toby@rhc-server ~]$ tftp -4 localhost
tftp> get this_is_a_test
tftp> quit
[toby@rhc-server ~]$ ls
this_is_a_test
[toby@rhc-server ~]$ cat this_is_a_test 
Hello?  Is it me you're looking for?

Looking good so far! 🙂

Testing remotely from client

Before we can test lets determine if the tftp package is installed, in my case it wasn’t, so I installed it;

[toby@rhc-client ~]$ yum search tftp
Loaded plugins: fastestmirror, langpacks
Determining fastest mirrors
============================================================ N/S matched: tftp =============================================================
syslinux-tftpboot.x86_64 : SYSLINUX modules in /tftpboot, available for network booting
tftp.x86_64 : The client for the Trivial File Transfer Protocol (TFTP)
tftp-server.x86_64 : The server for the Trivial File Transfer Protocol (TFTP)

  Name and summary matches only, use "search all" for everything.
[toby@rhc-client ~]$ yum list tftp
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Available Packages
tftp.x86_64                                                     5.2-11.el7                                                     th_lab_server
[toby@rhc-client ~]$ sudo yum install tftp
[sudo] password for toby: 
Loaded plugins: fastestmirror, langpacks
th_lab_server                                                                                                        | 3.6 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package tftp.x86_64 0:5.2-11.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                      Arch                           Version                            Repository                             Size
============================================================================================================================================
Installing:
 tftp                         x86_64                         5.2-11.el7                         th_lab_server                          35 k

Transaction Summary
============================================================================================================================================
Install  1 Package

Total download size: 35 k
Installed size: 48 k
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/th_lab_server/packages/tftp-5.2-11.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for tftp-5.2-11.el7.x86_64.rpm is not installed
tftp-5.2-11.el7.x86_64.rpm                                                                                           |  35 kB  00:00:00     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-0.1406.el7.centos.2.3.x86_64 (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : tftp-5.2-11.el7.x86_64                                                                                                   1/1 
  Verifying  : tftp-5.2-11.el7.x86_64                                                                                                   1/1 

Installed:
  tftp.x86_64 0:5.2-11.el7                                                                                                                  

Complete!

And now the test from the client.

[toby@rhc-client ~]$ sudo firewall-cmd --zone public --add-service tftp
[sudo] password for toby: 
success
[toby@rhc-client ~]$ tftp rhc-server
tftp> get this_is_a_test
tftp> quit
[toby@rhc-client ~]$ ls
Desktop  Documents  Downloads  Music  Pictures  Public  Templates  test  this_is_a_test  Videos
[toby@rhc-client ~]$ cat this_is_a_test 
Hello?  Is it me you're looking for?
[toby@rhc-client ~]$ sudo firewall-cmd --zone public --remove-service tftp
success

Note. If you don’t temporarily enable the tftp port on the client, the test will fail. I got the follow error via tcpdump which highlighted that the firewall was blocking the request;

23:25:40.388650 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 54)
    192.168.20.50.36588 > 192.168.20.1.69: [udp sum ok]  26 RRQ "this_is_a_test" netascii
23:25:40.390312 IP (tos 0x0, ttl 64, id 31747, offset 0, flags [none], proto UDP (17), length 70)
    192.168.20.1.43788 > 192.168.20.50.36588: [bad udp cksum 0xa9c7 -> 0xddd1!] UDP, length 42
23:25:40.390721 IP (tos 0xc0, ttl 64, id 3881, offset 0, flags [none], proto ICMP (1), length 98)
    192.168.20.50 > 192.168.20.1: ICMP host 192.168.20.50 unreachable - admin prohibited, length 78
	IP (tos 0x0, ttl 64, id 31747, offset 0, flags [none], proto UDP (17), length 70)
    192.168.20.1.43788 > 192.168.20.50.36588: [udp sum ok] UDP, length 42

At this point we have now completed the necessary steps to ensure we have a working tftp service.

Setting up PXE

The second part of today’s post covers the steps needed to enable booting from the network to facilitate building machines without the hassle of creating USB bootable images or burning ISO images to CD or DVD.

Out task list for this section is;

  • Add the additional config to the DHCP scope
  • Ensure we have the pxelinux/syslinux installed and copied to the required location
  • Create a basic menu to provide end users with a installation options
  • Test

Configuring DHCP

So, lets start by reminding ourselves what the current dhcpd.conf file looks like;

[toby@rhc-server ~]$ sudo cat /etc/dhcp/dhcpd.conf
[sudo] password for toby: 
#
#  lab.tobyheywood.com dhcp daemon configuration file
#
#  2016-02-22 - Initial creation
#

# Define which IP to listen on.  NOTE. daemon can only listen to one
# IP at a time if defined.
local-address 192.168.20.1;

# option definitions common to all supported networks...
option domain-name "lab.tobyheywood.com";
option domain-name-servers ns.lab.tobyheywood.com;

default-lease-time 600;
max-lease-time 7200;

# Use this to enble / disable dynamic dns updates globally.
#ddns-update-style interim;

# This is the authoritative DHCP server.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# Interface which can be accessed from outside the sandbox
# **** NOT IN USE ****
subnet 192.168.122.0 netmask 255.255.255.0 {
}

# The lab network
subnet 192.168.20.0 netmask 255.255.255.128 {
  range 192.168.20.50 192.168.20.99;
  option routers rtr.lab.tobyheywood.com;
}

So in order to get PXE (Preboot eXecution Environment) to function we will need to add a few more lines to the configuration, as highlighted below.

[toby@rhc-server ~]$ sudo cat /etc/dhcp/dhcpd.conf
[sudo] password for toby: 
#
#  lab.tobyheywood.com dhcp daemon configuration file
#
#  2016-02-22 - Initial creation
#

# Define which IP to listen on.  NOTE. daemon can only listen to one
# IP at a time if defined.
local-address 192.168.20.1;

# option definitions common to all supported networks...
option domain-name "lab.tobyheywood.com";
option domain-name-servers ns.lab.tobyheywood.com;

default-lease-time 600;
max-lease-time 7200;

# Use this to enble / disable dynamic dns updates globally.
#ddns-update-style interim;

# This is the authoritative DHCP server.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# Interface which can be accessed from outside the sandbox
# **** NOT IN USE ****
subnet 192.168.122.0 netmask 255.255.255.0 {
}

# The lab network
subnet 192.168.20.0 netmask 255.255.255.128 {
  range 192.168.20.50 192.168.20.99;
  option routers rtr.lab.tobyheywood.com;
}

# Additional configuration for PXE booting
allow booting;
allow bootp;
option option-128 code 128 = string;
option option-129 code 129 = text;
next-server 192.168.20.1;
filename "/pxelinux.0";

Setting up the required syslinux files in /tftpboot

During the section above when testing the tftp service on the client, I saw that there may be a shortcut to getting things up and running.  Everything I have read says you need to manually copy the syslinux files into the /tftpboot directory, however, if you search the rpm database with yum for anything tftp related, I see that there is potentially an easier way.

[toby@rhc-server ~]$ yum list tftp
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Installed Packages
tftp.x86_64                                     5.2-11.el7                                     @baselocal
[toby@rhc-server ~]$ yum search tftp
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
=========================================== N/S matched: tftp ===========================================
syslinux-tftpboot.x86_64 : SYSLINUX modules in /tftpboot, available for network booting
tftp.x86_64 : The client for the Trivial File Transfer Protocol (TFTP)
tftp-server.x86_64 : The server for the Trivial File Transfer Protocol (TFTP)

As you can see there appears to be a syslinux-tftpboot rpm.  So lets see what it gives us;

[toby@rhc-server ~]$ sudo yum install syslinux-tftpboot
[sudo] password for toby: 
Loaded plugins: fastestmirror
baselocal                                                                         | 3.6 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package syslinux-tftpboot.x86_64 0:4.05-8.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================
 Package                        Arch                Version                 Repository              Size
=========================================================================================================
Installing:
 syslinux-tftpboot              x86_64              4.05-8.el7              baselocal              425 k

Transaction Summary
=========================================================================================================
Install  1 Package

Total download size: 425 k
Installed size: 1.3 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : syslinux-tftpboot-4.05-8.el7.x86_64                                                   1/1 
  Verifying  : syslinux-tftpboot-4.05-8.el7.x86_64                                                   1/1 

Installed:
  syslinux-tftpboot.x86_64 0:4.05-8.el7                                                                  

Complete!
[toby@rhc-server ~]$ ls /tftpboot/
cat.c32        dmitest.c32   host.c32     ls.c32       pcitest.c32   rosh.c32        vesamenu.c32
chain.c32      elf.c32       ifcpu64.c32  lua.c32      pmload.c32    sanboot.c32     vpdtest.c32
cmd.c32        ethersel.c32  ifcpu.c32    mboot.c32    poweroff.com  sdi.c32         whichsys.c32
config.c32     gfxboot.c32   ifplop.c32   memdisk      pwd.c32       sysdump.c32     zzjson.c32
cpuid.c32      gpxecmd.c32   int18.com    memdump.com  pxechain.com  this_is_a_test
cpuidtest.c32  gpxelinux.0   kbdmap.c32   meminfo.c32  pxelinux.0    ver.com
disk.c32       hdt.c32       linux.c32    menu.c32     reboot.c32    vesainfo.c32

It looks like we have a winner!  Now we just need to make sure that the Preboot environment has a kernel or two to play with;

[root@rhc-server ~]# cp /software/centos7/images/pxeboot/* /tftpboot/centos7/
[root@rhc-server ~]# ls -Z /tftpboot/centos7/
-rw-r--r--. root root unconfined_u:object_r:tftpdir_t:s0 initrd.img
-r--r--r--. root root unconfined_u:object_r:tftpdir_t:s0 TRANS.TBL
-rw-r--r--. root root unconfined_u:object_r:tftpdir_t:s0 upgrade.img
-rwxr-xr-x. root root unconfined_u:object_r:tftpdir_t:s0 vmlinuz

OK, so lets now create the directory which will hold the PXE menus;

[toby@rhc-server ~]$ sudo mkdir /tftpboot/pxelinux.cfg
[sudo] password for toby: 
[toby@rhc-server ~]$ ls -Zd /tftpboot/pxelinux.cfg
drwxr-xr-x. root root unconfined_u:object_r:tftpdir_t:s0 /tftpboot/pxelinux.cfg

And now lets look at what is required to create the menus from which we will be able to select installation options.

[root@rhc-server centos7]# cat /tftpboot/pxelinux.cfg/default 
DEFAULT menu.c32
PROMPT 0
TIMEOUT 300
ONTIMEOUT localdisk
MENU TITLE PXE Network Boot

LABEL localdisk
    MENU LABEL ^Local Hard Drive
    MENU DEFAULT
    LOCALBOOT 0

LABEL Install_CentOS_7_2
    MENU LABEL CentOS 7.2
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/isolinux/initrd.img  inst.repo=http://rhc-server.lab.tobyheywood.com/centos7

At this point, all I wanted to do was prove that the pxe settings in dhcp were correct and that maybe, just maybe, I could build a vm across the network, first time.  No such luck! 🙁  Well, I guess 50% of the way there.

You can skip the next and jump to here if you don’t want to understand the pain I went through to get this operational.

I thought things were going well.  I created a blank VM, made sure to select that I was going to boot from the network to install the OS in Virtual Manager and turned the VM on.

pxeboot_initial_screen

So far so good!

pxeboot_menu

I’m now feel pretty happy with myself.  I select CentOS 7.0 and hit the enter key… and then am presented with the expected Welcome to CentOS 7 screen, from where I can kick off a manual installation.

pxeboot_centos7_initial_install_screen

So all, in all, things look good.  However there is more work to be done, as the next step is to create a kickstart file to define how the base install should look.

Reference Material

Trivial FTP (tftp)

PXE

Featured image taken by Ed Robinson who kindly uploaded it to Flickr.com.  I have made no changes to this image and left it in its original form for your viewing pleasure.  And to give the page a bit of colour 😉

RHEL/CentOS 7 – Waiting for 1 threads to finish

Posted on 2 CommentsPosted in CentOS 7, Deployment, Fedora, Linux, RHEL 7

As with most of my posts recently I have been looking at what is required to setup an isolated lab environment and from what started out as a simple idea, has slightly snowballed, due to one or more issues along the way.

The most recent is… During the installation I found that after selecting my installation options via the GUI, I was confronted with a status message “Waiting for 1 threads to finish”.  It sat there for a very long time!

The fix

Adding the “inst.geoloc=0” kernel parameter.  So my CentOS 7.2 code block in the pxelinux.cfg/default now looks like this;

LABEL Install_CentOS_7_2
    MENU LABEL CentOS 7.2
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/images/pxeboot/initrd.img inst.repo=http://rhc-server.lab.tobyheywood.com/centos7 inst.geoloc=0

Why does this make a difference

Well, it turns out the RHEL/CentOS 7 installation process & Fedoras’ for that matter try to be clever and determine your geographical location.  Now in a sandboxed world, that is going to be pretty challenging for the installer to accomplish, so it sits there, doing for all intents and purposes not a lot.

Adding the above mentioned parameter helps by simply disabling that functionality.

As with all things in life reading the manual is very useful. 🙂

And here is the manual; https://rhinstaller.github.io/anaconda/boot-options.html

 

Thanks to Matt Brown on Flickr.com for the featured image

Warning /dev/root does not exist

Posted on Leave a commentPosted in CentOS 7, Deployment, Linux, RHEL 7

Whilst writing a post on setting up a Trivial FTP server and the PXE boot functionality within ISC dhcpd, I stumbled across a bit of an issue which prevented me from successfully deploying either CentOS 7 or RHEL 7 to my KVM virtual machine.

I got the following error;

pxeboot_emergency_mode

the key lines here being;

[  OK  ] Reached target Basic System.

which just sat there for a long time before I saw this output;

dracut-initqueue[536]: Warning: Could not boot.
dracut-initqueue[536]: Warning: /dev/root does not exist

I tried a number of different things to work through this issue, including;

  • Switching from CentOS 7 to RHEL 7 (in the hope that a supported product might fare better
  • Changing dhcpd configuration to try different ways of defining the PXE service
  • Experimenting with different parameters in the pxelinux.cfg/default file.
  • “Googling” it to death!

Now what I found after googling is that there are some additional parameters that (it would appear) MUST be set!!  Things have changed a lot since the days of RHEL 6/CentOS 6.

For me I had to add the following “inst.repo=http://rhc-server.lab.tobyheywood.com/centos7” to the APPEND line for the kernel I intended to boot from.  This essentially defines where the installation files are in the absence of a kickstart aka anaconda file!  This is the important bit!  Because I was intend on proving the manual installation would work without have to have the physical media in my hand, I had skipped the kickstart file.

When you think about it this makes perfect sense, as otherwise how would it know where the installation files are???  Exactly!

So my pxelinux.cfg/default file now looks like this;

[toby@rhc-server rhel7]$ sudo cat /tftpboot/pxelinux.cfg/default 
DEFAULT menu.c32
PROMPT 0
TIMEOUT 300
ONTIMEOUT localdisk
MENU TITLE PXE Network Boot

LABEL localdisk
    MENU LABEL ^Local Hard Drive
    MENU DEFAULT
    LOCALBOOT 0

LABEL Install_CentOS_7_2_pxeboot
    MENU LABEL CentOS 7.2
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/images/pxeboot/initrd.img inst.repo=http://rhc-server.lab.tobyheywood.com/centos7

For those that want some additional bedtime reading, I would highly recommend the following the Anaconda documentation that relates to boot options – https://rhinstaller.github.io/anaconda/boot-options.html

SQL Server loves Linux

Breaking news – Microsoft SQL Server for Linux?

Posted on Leave a commentPosted in Linux, Microsoft, RHEL 7, System Administration

If we are to believe the “official” Microsoft blog and the raft of other news sites that have written about this in the past 24 hours, then Microsoft is continuing to change direction.

It has announced that it is targeting mid 2017 for a release of SQL Server which will run on Linux.  And as of today there is a private preview release of SQL Server for Linux.  Though I suspect you need to be the right kind of customer to gain access to that.

This to me, seems fairly unsurprising really.  Especially when you consider the Microsoft and Red Hat have been developing a “ever closer union” (a.k.a. relationship) over the past year or so.  And also that Microsoft has been maintaining Skype clients on Linux and their efforts around office on the Mac OS X (which is derived from *nix).

Will be interesting to see what else gets ported from the Microsoft campus into the realm of Linux.  I would also be interested to see what Oracle think of this.

You are pretty much guaranteed that the Linux camp are going to be split, there will be those who are actively against any form of proprietary software (even when the software is still contained in a shrink wrapped box, next to a machine running Linux).  And those who don’t really have an issue with proprietary software on Linux.

I guess I’m probably in the minority, but if this means that enterprise is able to increase the adoption of Linux at the OS level, and to get the benefits of using Linux in the enterprise space, without having to re-engineer all their past works, then that is a good thing.

I suspect, that SQL Server administration will still be a completely administered via the SQL Server Management Studio or T-SQL, and so the DBA’s for the most part will not have to care to much about the underlying infrastructure.  The Linux SAs can continue to work in their world too.

But don’t get me wrong, I’m sure there will be some pain for both DBAs and SAs, but then if nothing else it will make for some interesting conversations around the coffee machine.  And I’m sure that if you want/need Mixed Mode authentication then there will be a lot of work to make sure your Linux servers are members of an Active Directory domain.

 

Picture of fiber connected switches and servers

Back to basics – Creating a centralised yum/dnf repository

Posted on Leave a commentPosted in CentOS 7, Fedora, Linux, RHEL 7, System Administration

So far in the “Back to Basics” series (if you can call it that), I’ve covered, setting up a local yum repository, creating a internal DNS server and creating a DHCP server.  Oh, and also then correcting the fact that I had missed the reverse DNS zone for my lab network!  Doh!!!  Now, none of this was by accident (accept the reverse zone mishap).

In an isolated network, access to installation media can be essential, DNS and DHCP a pretty much standard in all environments (there are some exceptions) and all are pretty much mandatory in order to get your network up and running.

The ultimate aim of this series is to end up with a server which can be used to build more servers and/or clients into the lab network that I am setting up.

Before we can reach this goal, there are a few outstanding things to tackle;

  • Making our local repository readable from within our network (this article)
  • Setting up a TFTP server and confirm it works
  • Enable PXE booting functionality via DHCPd
  • Customising our installs using Kickstart

The above will then provide a basic but functional method of deploying more servers and clients into the lab environment, across the network and removes the need for monkeying around with ISO images, USB sticks (if you were to do similar in a real network) and once tested removes the human errors that can be introduced when manually installing an OS multiple times.

So what do we need

  • Apache (a.k.a. httpd)

Installing Apache

Given that I set up a local yum repository based on the installation media, it couldn’t be simpler.

[toby@rhc-server ~]$ sudo yum install httpd<br />
Loaded plugins: fastestmirror<br />
baselocal                                                                                                            | 3.6 kB  00:00:00     <br />
Loading mirror speeds from cached hostfile<br />
Resolving Dependencies<br />
--&gt; Running transaction check<br />
---&gt; Package httpd.x86_64 0:2.4.6-17.el7.centos.1 will be installed<br />
--&gt; Processing Dependency: httpd-tools = 2.4.6-17.el7.centos.1 for package: httpd-2.4.6-17.el7.centos.1.x86_64<br />
--&gt; Processing Dependency: /etc/mime.types for package: httpd-2.4.6-17.el7.centos.1.x86_64<br />
--&gt; Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-17.el7.centos.1.x86_64<br />
--&gt; Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-17.el7.centos.1.x86_64<br />
--&gt; Running transaction check<br />
---&gt; Package apr.x86_64 0:1.4.8-3.el7 will be installed<br />
---&gt; Package apr-util.x86_64 0:1.5.2-6.el7 will be installed<br />
---&gt; Package httpd-tools.x86_64 0:2.4.6-17.el7.centos.1 will be installed<br />
---&gt; Package mailcap.noarch 0:2.1.41-2.el7 will be installed<br />
--&gt; Finished Dependency Resolution</p>
<p>Dependencies Resolved</p>
<p>============================================================================================================================================<br />
 Package                         Arch                       Version                                     Repository                     Size<br />
============================================================================================================================================<br />
Installing:<br />
 httpd                           x86_64                     2.4.6-17.el7.centos.1                       baselocal                     2.7 M<br />
Installing for dependencies:<br />
 apr                             x86_64                     1.4.8-3.el7                                 baselocal                     103 k<br />
 apr-util                        x86_64                     1.5.2-6.el7                                 baselocal                      92 k<br />
 httpd-tools                     x86_64                     2.4.6-17.el7.centos.1                       baselocal                      77 k<br />
 mailcap                         noarch                     2.1.41-2.el7                                baselocal                      31 k</p>
<p>Transaction Summary<br />
============================================================================================================================================<br />
Install  1 Package (+4 Dependent packages)</p>
<p>Total download size: 3.0 M<br />
Installed size: 10 M<br />
Is this ok [y/d/N]: y<br />
Downloading packages:<br />
--------------------------------------------------------------------------------------------------------------------------------------------<br />
Total                                                                                                        12 MB/s | 3.0 MB  00:00:00     <br />
Running transaction check<br />
Running transaction test<br />
Transaction test succeeded<br />
Running transaction<br />
  Installing : apr-1.4.8-3.el7.x86_64                                                                                                   1/5<br />
  Installing : apr-util-1.5.2-6.el7.x86_64                                                                                              2/5<br />
  Installing : httpd-tools-2.4.6-17.el7.centos.1.x86_64                                                                                 3/5<br />
  Installing : mailcap-2.1.41-2.el7.noarch                                                                                              4/5<br />
  Installing : httpd-2.4.6-17.el7.centos.1.x86_64                                                                                       5/5<br />
  Verifying  : mailcap-2.1.41-2.el7.noarch                                                                                              1/5<br />
  Verifying  : httpd-2.4.6-17.el7.centos.1.x86_64                                                                                       2/5<br />
  Verifying  : apr-util-1.5.2-6.el7.x86_64                                                                                              3/5<br />
  Verifying  : apr-1.4.8-3.el7.x86_64                                                                                                   4/5<br />
  Verifying  : httpd-tools-2.4.6-17.el7.centos.1.x86_64                                                                                 5/5 </p>
<p>Installed:<br />
  httpd.x86_64 0:2.4.6-17.el7.centos.1                                                                                                      </p>
<p>Dependency Installed:<br />
  apr.x86_64 0:1.4.8-3.el7   apr-util.x86_64 0:1.5.2-6.el7   httpd-tools.x86_64 0:2.4.6-17.el7.centos.1   mailcap.noarch 0:2.1.41-2.el7  </p>
<p>Complete!

Confirm file and folder permissions

[toby@rhc-server ~]$ ll -Z /software/<br />
drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 centos7<br />

One thing to consider here is SELinux.

Before you run for the hills screaming, just take a deep breath and embrace something that by default will make your server more secure, yes it does have a bit of a learning curve but doesn’t everything?

I’m a believer in using the tools available to ensure I end up with a secure and stable environment.  SELinux is one of those things which for many years I avoided like the plague but to be honest, that was due to me not having had the time to properly understand what it does and how it does it.

After spending some time tinkering with it, it didn’t seem half as scary.  For sure, it complicates things a little when you come to troubleshoot permission issues, but then everything is more contained.

Lets make sure that the directory containing the installation media, which is in a none standard location (as far as Apache is concerned), has the correct SELinux permissions assigned to the folder structure.  The easiest way is to copy the existing SELinux contexts from the /var/www/html and here is the command to do that;

[root@rhc-server ~]# cd /var/www/<br />
[root@rhc-server www]# ll -Z<br />
drwxr-xr-x. root root system_u:object_r:httpd_sys_script_exec_t:s0 cgi-bin<br />
drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 html<br />
[root@rhc-server www]# chcon -R --reference=/var/www/html/ /software/centos7<br />
[root@rhc-server www]# ll -Z /software/<br />
drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 centos7<br />

As you can see we now have the right context associated with the centos7 directory, now we need to make sure the httpd.conf file is updated to present the centos7 directory and it’s contents to the outside world.

Rather than modifying the httpd.conf file itself it is recommended that you create your own .conf files in /etc/httpd/conf.d/ and these will be loaded after the initial httpd.conf file.  I created a single test files as follows;

[toby@rhc-server ~]$ cat /etc/httpd/conf.d/software.conf<br />
Alias &quot;/centos7&quot; &quot;/software/centos7&quot;</p>
<p>&lt;Directory /software/centos7&gt;<br />
Options +Indexes</p>
<p>Order allow,deny<br />
Allow from all<br />
Require all granted<br />
&lt;/Directory&gt;<br />

Screenshot showing successful directory listing from client machine of centos7 media
Screenshot showing successful directory listing from client machine of centos7 media

The Alias allows you to point to a directory which is outside of Apaches’ DocumentRoot, (typically set to /var/www/html).  The Directory block, contains two things of note.  First, for testing I have added the “Options +Indexes” so that when I try to connect from a web browser on my client machine, I can confirm that I can see the contents of the repository directory.  The second chunk of config, starting “Order all,deny…” is there so that Apache will allow connections to this none standard location.

One thing I did have to do, that I haven’t stated above is allow HTTP connections through the firewall.

This was accomplished by way of a simple one liner;

[toby@rhc-server ~]$ sudo firewall-cmd --zone=public --add-service=http

Note.  To make this new firewall rule permanent you need to use the “–permanent” firewall-cmd option on the command line.  I added this afterwards once I was happy that everything was working.

Configuring a yum .repo file to access the centralised software repository

This is very similar in the steps taken when I setup the local yum repository.  The only difference this time will be that I’ll give it a more meaningful name and the file location will be a http:// address rather that a file:///.

So here is what I have put together;

[root@rhc-client yum.repos.d]# cat CentOS-lab-Media.repo<br />
[th_lab_server]<br />
baseurl=http://rhc-server.lab.tobyheywood.com/centos7/<br />
gpgcheck=1<br />
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7<br />

And then, as the saying goes, the proof is in the pudding;

[root@rhc-client yum.repos.d]# yum repolist<br />
Loaded plugins: fastestmirror, langpacks<br />
Repository 'th_lab_server' is missing name in configuration, using id<br />
th_lab_server                                                                                                        | 3.6 kB  00:00:00     <br />
(1/2): th_lab_server/group_gz                                                                                        | 157 kB  00:00:00     <br />
(2/2): th_lab_server/primary_db                                                                                      | 4.9 MB  00:00:00     <br />
Loading mirror speeds from cached hostfile<br />
repo id                                                            repo name                                                          status<br />
th_lab_server                                                      th_lab_server                                                      8,465<br />
repolist: 8,465<br />

Oops, now it would appear I haven’t added a name parameter in the dot repo file.  Let me correct that…

[root@rhc-client yum.repos.d]# cat CentOS-lab-Media.repo<br />
[th_lab_server]<br />
name=&quot;CentOS7 Media on rhc-server.lab.tobyheywood.com&quot;<br />
baseurl=http://rhc-server.lab.tobyheywood.com/centos7/<br />
gpgcheck=1<br />
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7<br />

And now if I run the command “yum repolist” again, it should return the list of enabled repositories without complaining, oh and the repo name column will also show my desired name for my network enabled repository ( a shorter name may be better);

[root@rhc-client yum.repos.d]# yum repolist<br />
Loaded plugins: fastestmirror, langpacks<br />
Loading mirror speeds from cached hostfile<br />
repo id                                          repo name                                                              status<br />
th_lab_server                                    &quot;CentOS7 Media on rhc-server.lab.tobyheywood.com&quot;             8,465<br />
repolist: 8,465

And there we have it a working centralised repository, if you don’t have access to Red Hat Satellite server or if you don’t want to install the open source version Spacewalk.

I guess the final test, would be to install a couple of packages;

[root@rhc-client yum.repos.d]# yum install iostat<br />
Loaded plugins: fastestmirror, langpacks<br />
th_lab_server                                                                                                        | 3.6 kB  00:00:00<br />
Loading mirror speeds from cached hostfile<br />
No package iostat available.<br />
Error: Nothing to do<br />
[root@rhc-client yum.repos.d]# yum whatprovides iostat<br />
Loaded plugins: fastestmirror, langpacks<br />
Loading mirror speeds from cached hostfile<br />
th_lab_server/filelists_db                                                                                           | 5.8 MB  00:00:00<br />
sysstat-10.1.5-4.el7.x86_64 : Collection of performance monitoring tools for Linux<br />
Repo        : th_lab_server<br />
Matched from:<br />
Filename    : /usr/bin/iostat</p>
<p>sysstat-10.1.5-4.el7.x86_64 : Collection of performance monitoring tools for Linux<br />
Repo        : @anaconda<br />
Matched from:<br />
Filename    : /usr/bin/iostat</p>
<p>[root@rhc-client yum.repos.d]# yum install sysstat -y<br />
Loaded plugins: fastestmirror, langpacks<br />
Loading mirror speeds from cached hostfile<br />
Package sysstat-10.1.5-4.el7.x86_64 already installed and latest version<br />
Nothing to do<br />

Ah, I guess a demo is never meant to go really smoothly, but this is probably better as it demonstrates some awesome functionality that yum has.

Line 3, shows that it has found my networked repo, line 5 shows that there is no such package in the repo, line 7 highlights a way to find the right package for the command you want to run, lines 10 through 14 show conclusively that it has connected across the network to the repo, and has found a suitable package called sysstat. In line 21, I’m trying to install the package only to be told in line 24 that it’s already installed.

The really keen eyed of you may have also spotted the @anaconda repo, this should have rung an alarm bell in my head to say, hey!  What are you doing?  Its already installed!

Useful link – RPM DB Recovery

Posted on Leave a commentPosted in CentOS 7, Fedora, Linux, RHEL 6, RHEL 7, RPM, System Administration

Every now and then, we find ourselves in a bit of a predicament.  In this instance whilst performing an upgrade on a server, things just weren’t going well and it appeared we had some corruption in the RPM database on one of our servers.

We were seeing segmentation faults when trying to use “rpm”.

The following page on the rpm.org website proved very useful, in getting things back up and running swiftly;

http://www.rpm.org/wiki/Docs/RpmRecovery

DHCPD and the mysterious “host unknown”

Posted on Leave a commentPosted in CentOS 7, DHCP, DNS, Fedora, Linux, RHEL 7

For those of you following my “back to basics” series, in my last post setting up a dhcp server, I finished up with an error that flummoxed me initially.

Feb 22 22:13:36 rhc-server dhcpd[2682]: ns.lab.tobyheywood.com: host unknown.<br />
Feb 22 22:13:36 rhc-server dhcpd[2682]: rtr.lab.tobyheywood.com: host unknown.

I had tried;

  • pinging the supposedly unknown hosts
    • both locally on the server which has the named daemon on it
    • remotely from the client that had just been assigned an IP address from my newly installed dhcp server
  • used tools such as dig to confirm the zone looked fine
  • double checked the dhcpd.conf file
  • nothing stood out as being odd
  • So, as it was late, I went to bed with the determination that I would research and fix the issue before me!

So I started reading through the dhcpd.conf man page (an online version though no guarantee it’s the latest version, can be found here http://linux.die.net/man/5/dhcpd.conf).

After scanning through the first few sections, I began reading about the steps required to setup Dynamic DNS Updates (ddns) and spied the recommendations regarding securing the dynamic DNS configuration. And something suddenly occurred to me… I hadn’t created a reverse lookup zone!!!

To the bat cave!

Now before I begin, there are two things I should mention;

  1. It’s good to reboot
  2. After rebooting my virtual machines it fixed my issue, but I suspect something had not refreshed as both the DNS and DHCP servers were created in the same evening.  I had also made modifications to my NIC setup to reflect the fact that I now had a local DNS server.
  3. Bonus material, I shall continue with the reverse zone creation steps.

Just to double check that I am setting up my reverse zone, I also consulted the current documentation for Red Hat Enterprise Linux 7 (essential CentOS 7) https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-BIND.html#sec-bind-zone.  I guess now is a good time to mention the fact that Red Hat does have some awesome documentation and you really should take the time to read all of it (which I have), as quite often not only do you find out the bits of information you were looking for but also you will find some interesting shortcuts and new ways of doing things.

Anyway…

Creating a reverse zone

So my reverse zone file looks like this

[root@rhc-server data]# cat lab.tobyheywood.com.rr.zone<br />
$ORIGIN 20.168.192.in-addr.arpa.<br />
$TTL 1d<br />
@    IN    SOA    ns.lab.tobyheywood.com.    hostmaster.lab.tobyheywood.com. (<br />
2016022400    ; Serial<br />
6h        ; Refresh<br />
1h        ; Retry<br />
2d        ; Expire<br />
1d )        ; Minimum TTL<br />
;<br />
@    IN    NS    ns.lab.tobyheywood.com.<br />
;<br />
1    IN    PTR    rhc-server.lab.tobyheywood.com.<br />

Annoyingly the formatting seems to ignore tabs and multiple spacing where I put the code snippets on my website but I can assure you it is all neatly indented to ensure easy reading.

And I also added the following into the /etc/named.conf file;

[root@rhc-server data]# cat /etc/named.conf<br />
options {<br />
listen-on port 53 { 192.168.20.1; };<br />
//listen-on-v6 port 53 { ::1; };<br />
directory     &quot;/var/named&quot;;<br />
dump-file     &quot;/var/named/data/cache_dump.db&quot;;<br />
statistics-file &quot;/var/named/data/named_stats.txt&quot;;<br />
memstatistics-file &quot;/var/named/data/named_mem_stats.txt&quot;;<br />
allow-query     { any; };</p>
<p>dnssec-enable yes;<br />
dnssec-validation yes;<br />
dnssec-lookaside auto;</p>
<p>/* Path to ISC DLV key */<br />
bindkeys-file &quot;/etc/named.iscdlv.key&quot;;</p>
<p>managed-keys-directory &quot;/var/named/dynamic&quot;;</p>
<p>pid-file &quot;/run/named/named.pid&quot;;<br />
session-keyfile &quot;/run/named/session.key&quot;;<br />
};</p>
<p>logging {<br />
channel default_debug {<br />
file &quot;data/named.run&quot;;<br />
severity dynamic;<br />
};<br />
};</p>
<p>zone &quot;.&quot; IN {<br />
type hint;<br />
file &quot;named.ca&quot;;<br />
};</p>
<p>zone &quot;lab.tobyheywood.com&quot; IN {<br />
type master;<br />
file &quot;data/lab.tobyheywood.com&quot;;<br />
allow-update { none; };<br />
};</p>
<p>zone &quot;20.168.192.in-addr.arpa&quot; IN {<br />
type master;<br />
file &quot;data/lab.tobyheywood.com.rr.zone&quot;;<br />
allow-update { none; };<br />
};</p>
<p>include &quot;/etc/named.rfc1912.zones&quot;;<br />
include &quot;/etc/named.root.key&quot;;<br />

Now I spotted that “recursion yes” had not be commented whilst I was adding the new zone “20.168.192.in-addr.arpa”.  This is unwanted as I do not want the server to be a caching DNS server, I’m only interested at this time in it providing my internal host details and nothing more.

[root@rhc-server data]# dig -x 192.168.20.1</p>
<p>; &lt;&lt;&gt;&gt; DiG 9.9.4-RedHat-9.9.4-14.el7 &lt;&lt;&gt;&gt; -x 192.168.20.1<br />
;; global options: +cmd<br />
;; Got answer:<br />
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 45831<br />
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2</p>
<p>;; OPT PSEUDOSECTION:<br />
; EDNS: version: 0, flags:; udp: 4096<br />
;; QUESTION SECTION:<br />
;1.20.168.192.in-addr.arpa.    IN    PTR</p>
<p>;; ANSWER SECTION:<br />
1.20.168.192.in-addr.arpa. 86400 IN    PTR    rhc-server.lab.tobyheywood.com.</p>
<p>;; AUTHORITY SECTION:<br />
20.168.192.in-addr.arpa. 86400    IN    NS    ns.lab.tobyheywood.com.</p>
<p>;; ADDITIONAL SECTION:<br />
ns.lab.tobyheywood.com.    86400    IN    A    192.168.20.1</p>
<p>;; Query time: 0 msec<br />
;; SERVER: 192.168.20.1#53(192.168.20.1)<br />
;; WHEN: Wed Feb 24 23:20:47 GMT 2016<br />
;; MSG SIZE  rcvd: 131<br />

And there we have it, a reverse DNS zone has been born!  Yay!