Back to basics – Kickstart your anaconda file (your systems blueprint)

Posted on Leave a commentPosted in CentOS 7, Deployment, Linux, RHEL 7, System Administration

The final piece in this jigsaw puzzle of network installation madness is the kickstart file.  It is the blueprint from which your RHEL/CentOS/Fedora/[enter distribution name here] is built from.

My preferred way of creating the initial template is to perform a manual installation and then to tweak the resultant /root/anaconda-ks.cfg file to my needs.  This is also the recommended method in the Red Hat documentation.  By doing it this way you can avoid some of the pitfalls of parameter changes within the kickstart script which may alter between releases of RHEL/CentOS.

So, here is what mine currently looks like;

[root@rhc-server ~]# cat anaconda-ks.cfg 
#version=RHEL7
# System authorization information
auth --enableshadow --passalgo=sha512

# Use CDROM installation media
cdrom
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='gb'
# System language
lang en_GB.UTF-8

# Network information
network  --bootproto=dhcp --device=ens3 --ipv6=auto --activate
network  --hostname=rhc-server
# Root password
rootpw --iscrypted $6$1RcgxNlQgKVJAEHj$dcQtZ3Jhe8vw1Aj3rB8zyLl2tTumL88czrcJo4nyUeashp7/rJvE3lFOmWsu9Ml.AZw7PSx5u1M0IGeTLa5ds.
# System timezone
timezone Europe/London --isUtc
user --groups=wheel --name=toby --password=$6$wRgpQD/lrXaqF8cW$PdCIE3CdBq5PxWYejCSpWFVuiMGUIs.eKQXPR5pDeHTmKp3/10qazLDVaCJcpp7zenDKWNqWPXrYjEGRJsAK41 --iscrypted --gecos="toby"
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
autopart --type=lvm
# Partition clearing information
clearpart --none --initlabel 

%packages
@core

%end

Over time, the kickstart file will evolve, and it is a wise man or woman who validates the configuration before actually trying to use the file.  But, having said that, the ksvalidator utility (as the documentation states), can only validate things so far and it will not look at the %pre, %post or %packages sections.  It also doesn’t guarantee a successful install, it just provides a sanity check before you really test it.

So anyway, here is my slightly modified version of the kickstart file, which I have saved in a directory which is accessible via the web server.

[root@rhc-server ks]# cat /var/www/html/ks/basic.ks 
# System authorization information
auth --enableshadow --passalgo=sha512

# Use installation files via http
install
url --url=http://rhc-server.lab.tobyheywood.com/centos7/

# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda

# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='gb'

# System language
lang en_GB.UTF-8

# Network information
network  --bootproto=dhcp --device=ens3 --ipv6=auto --activate

# Root password
rootpw --iscrypted $6$1RcgxNlQgKVJAEHj$dcQtZ3Jhe8vw1Aj3rB8zyLl2tTumL88czrcJo4nyUeashp7/rJvE3lFOmWsu9Ml.AZw7PSx5u1M0IGeTLa5ds.

# System timezone
timezone Europe/London --isUtc
user --groups=wheel --name=toby --password=$6$wRgpQD/lrXaqF8cW$PdCIE3CdBq5PxWYejCSpWFVuiMGUIs.eKQXPR5pDeHTmKp3/10qazLDVaCJcpp7zenDKWNqWPXrYjEGRJsAK41 --iscrypted --gecos="toby"

# System bootloader configuration
bootloader --location=mbr --boot-drive=vda  --iscrypted --password=grub.pbkdf2.sha512.10000.AD234D09B3DDE933C60C46AAA52B95DBB2D48D766E8DDD1852FBF373C8D0474401E4FD6D1D997B1D169B35C69D7776B3FCAC6A3BB6338B0910EB0899B0452BFE.531AE5C20F8CA36DB321770537C0C872B4670EB60F9461A85D2D429C36BAC80F12EBDCC85A6514889332B70BBA780285F84DFDCA57A3B92C3F9FA7387F9F59A0

autopart --type=lvm
# Partition clearing information
clearpart --none --initlabel 

# Firewall & SELinux settings
firewall --enabled --ssh
selinux --enforcing

# Create yum .repo file
repo --name=TheLab --baseurl=http://rhc-server.lab.tobyheywood.com/centos7/ --install

%packages
@core

%end

reboot

I have highlighted all lines that I have modified and will explain any I have added along the way.

So what did I change?

  • Line 6 & 7 – These two lines are required to specify where the install files are actually located.  Note this does have to be the same structure as the ISO (just in case you were wondering).
  • Line 11 – Just in case we have any other disks available to us, we focus the installation to use only the first virtIO disk it sees.  Another point to make here is that device naming is not guaranteed and the documentation dues make some alternative suggestions here.
  • Line 30 – I updated the boot-drive parameter so that it correctly reflected the hard drive device naming and also followed a best practice of implementing a password for the grub2 bootloader.
  • Line 37 & 38 – I explicitly enable selinux in enforcing mode and also set a very basic firewall rule
  • Line 41 – Rather than having to go in and modify my own yum .repo file you can use this option with the install flag to set this up for you

Not many changes but hopefully it provides a good idea of what is needed to get a working install fully operational.

Validating your kickstart script

As mentioned earlier, it is worth validating you kickstart script simply as a sanity check, this can be done as follows;

Installing the ksvalidtor app

[root@rhc-server ks]# yum install pykickstart
Loaded plugins: fastestmirror
baselocal                                                                                                            | 3.6 kB  00:00:00     
(1/2): baselocal/group_gz                                                                                            | 155 kB  00:00:00     
(2/2): baselocal/primary_db                                                                                          | 2.8 MB  00:00:00     
Determining fastest mirrors
Resolving Dependencies
--> Running transaction check
---> Package pykickstart.noarch 0:1.99.66.6-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                           Arch                         Version                               Repository                       Size
============================================================================================================================================
Installing:
 pykickstart                       noarch                       1.99.66.6-1.el7                       baselocal                       328 k

Transaction Summary
============================================================================================================================================
Install  1 Package

Total download size: 328 k
Installed size: 1.5 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : pykickstart-1.99.66.6-1.el7.noarch                                                                                       1/1 
  Verifying  : pykickstart-1.99.66.6-1.el7.noarch                                                                                       1/1 

Installed:
  pykickstart.noarch 0:1.99.66.6-1.el7                                                                                                      

Complete!

Running ksvalidator

Note.  If there is no output after running the command it is a good sign!  Just for good measure I also like to echo out the return code following the command just to give me that extra warm feeling :).

[root@rhc-server ks]# ksvalidator /var/www/html/ks/basic.ks 
[root@rhc-server ks]# echo $?
0

The last thing we need to do is make sure this will be used, or as a bare minimum, that it can be used.  So lets edit the pxelinux.cfg/default file;

[root@rhc-server ks]# cat /tftpboot/pxelinux.cfg/default 
DEFAULT menu.c32
PROMPT 0
TIMEOUT 300
ONTIMEOUT localdisk
MENU TITLE PXE Network Boot

LABEL localdisk
    MENU LABEL ^Local Hard Drive
    MENU DEFAULT
    LOCALBOOT 0

LABEL Install_CentOS_7_2
    MENU LABEL CentOS 7.2
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/images/pxeboot/initrd.img inst.repo=http://rhc-server.lab.tobyheywood.com/centos7 inst.geoloc=0

LABEL Install_CentOS_7_2_KS
    MENU LABEL CentOS 7.2 (KS)
    KERNEL centos7/vmlinuz
    APPEND initrd=http://rhc-server.lab.tobyheywood.com/centos7/images/pxeboot/initrd.img inst.ks=http://rhc-server.lab.tobyheywood.com/ks/basic.ks inst.geoloc=0

The key thing here is that I have done away with the “repo.inst” parameter and it’s associated value and replaced it with the  “inst.ks” parameter, which points to the kickstart file I created earlier.

And that is it, we have a very basic automated installation of CentOS/RHEL 7 over the network without having to pick up a single ISO image and burn it to disk, or USB stick, or manually mount it to a VM.

Reference Material

Credit to Will Scullin, who made his Blueprint image available on Flickr.com.

Picture of fiber connected switches and servers

Back to basics – Creating a centralised yum/dnf repository

Posted on Leave a commentPosted in CentOS 7, Fedora, Linux, RHEL 7, System Administration

So far in the “Back to Basics” series (if you can call it that), I’ve covered, setting up a local yum repository, creating a internal DNS server and creating a DHCP server.  Oh, and also then correcting the fact that I had missed the reverse DNS zone for my lab network!  Doh!!!  Now, none of this was by accident (accept the reverse zone mishap).

In an isolated network, access to installation media can be essential, DNS and DHCP a pretty much standard in all environments (there are some exceptions) and all are pretty much mandatory in order to get your network up and running.

The ultimate aim of this series is to end up with a server which can be used to build more servers and/or clients into the lab network that I am setting up.

Before we can reach this goal, there are a few outstanding things to tackle;

  • Making our local repository readable from within our network (this article)
  • Setting up a TFTP server and confirm it works
  • Enable PXE booting functionality via DHCPd
  • Customising our installs using Kickstart

The above will then provide a basic but functional method of deploying more servers and clients into the lab environment, across the network and removes the need for monkeying around with ISO images, USB sticks (if you were to do similar in a real network) and once tested removes the human errors that can be introduced when manually installing an OS multiple times.

So what do we need

  • Apache (a.k.a. httpd)

Installing Apache

Given that I set up a local yum repository based on the installation media, it couldn’t be simpler.

[toby@rhc-server ~]$ sudo yum install httpd
Loaded plugins: fastestmirror
baselocal                                                                                                            | 3.6 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.4.6-17.el7.centos.1 will be installed
--> Processing Dependency: httpd-tools = 2.4.6-17.el7.centos.1 for package: httpd-2.4.6-17.el7.centos.1.x86_64
--> Processing Dependency: /etc/mime.types for package: httpd-2.4.6-17.el7.centos.1.x86_64
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-17.el7.centos.1.x86_64
--> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-17.el7.centos.1.x86_64
--> Running transaction check
---> Package apr.x86_64 0:1.4.8-3.el7 will be installed
---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed
---> Package httpd-tools.x86_64 0:2.4.6-17.el7.centos.1 will be installed
---> Package mailcap.noarch 0:2.1.41-2.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                         Arch                       Version                                     Repository                     Size
============================================================================================================================================
Installing:
 httpd                           x86_64                     2.4.6-17.el7.centos.1                       baselocal                     2.7 M
Installing for dependencies:
 apr                             x86_64                     1.4.8-3.el7                                 baselocal                     103 k
 apr-util                        x86_64                     1.5.2-6.el7                                 baselocal                      92 k
 httpd-tools                     x86_64                     2.4.6-17.el7.centos.1                       baselocal                      77 k
 mailcap                         noarch                     2.1.41-2.el7                                baselocal                      31 k

Transaction Summary
============================================================================================================================================
Install  1 Package (+4 Dependent packages)

Total download size: 3.0 M
Installed size: 10 M
Is this ok [y/d/N]: y
Downloading packages:
--------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                        12 MB/s | 3.0 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : apr-1.4.8-3.el7.x86_64                                                                                                   1/5 
  Installing : apr-util-1.5.2-6.el7.x86_64                                                                                              2/5 
  Installing : httpd-tools-2.4.6-17.el7.centos.1.x86_64                                                                                 3/5 
  Installing : mailcap-2.1.41-2.el7.noarch                                                                                              4/5 
  Installing : httpd-2.4.6-17.el7.centos.1.x86_64                                                                                       5/5 
  Verifying  : mailcap-2.1.41-2.el7.noarch                                                                                              1/5 
  Verifying  : httpd-2.4.6-17.el7.centos.1.x86_64                                                                                       2/5 
  Verifying  : apr-util-1.5.2-6.el7.x86_64                                                                                              3/5 
  Verifying  : apr-1.4.8-3.el7.x86_64                                                                                                   4/5 
  Verifying  : httpd-tools-2.4.6-17.el7.centos.1.x86_64                                                                                 5/5 

Installed:
  httpd.x86_64 0:2.4.6-17.el7.centos.1                                                                                                      

Dependency Installed:
  apr.x86_64 0:1.4.8-3.el7   apr-util.x86_64 0:1.5.2-6.el7   httpd-tools.x86_64 0:2.4.6-17.el7.centos.1   mailcap.noarch 0:2.1.41-2.el7  

Complete!

Confirm file and folder permissions

[toby@rhc-server ~]$ ll -Z /software/
drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 centos7

One thing to consider here is SELinux.

Before you run for the hills screaming, just take a deep breath and embrace something that by default will make your server more secure, yes it does have a bit of a learning curve but doesn’t everything?

I’m a believer in using the tools available to ensure I end up with a secure and stable environment.  SELinux is one of those things which for many years I avoided like the plague but to be honest, that was due to me not having had the time to properly understand what it does and how it does it.

After spending some time tinkering with it, it didn’t seem half as scary.  For sure, it complicates things a little when you come to troubleshoot permission issues, but then everything is more contained.

Lets make sure that the directory containing the installation media, which is in a none standard location (as far as Apache is concerned), has the correct SELinux permissions assigned to the folder structure.  The easiest way is to copy the existing SELinux contexts from the /var/www/html and here is the command to do that;

[root@rhc-server ~]# cd /var/www/
[root@rhc-server www]# ll -Z
drwxr-xr-x. root root system_u:object_r:httpd_sys_script_exec_t:s0 cgi-bin
drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 html
[root@rhc-server www]# chcon -R --reference=/var/www/html/ /software/centos7
[root@rhc-server www]# ll -Z /software/
drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 centos7

As you can see we now have the right context associated with the centos7 directory, now we need to make sure the httpd.conf file is updated to present the centos7 directory and it’s contents to the outside world.

Rather than modifying the httpd.conf file itself it is recommended that you create your own .conf files in /etc/httpd/conf.d/ and these will be loaded after the initial httpd.conf file.  I created a single test files as follows;

[toby@rhc-server ~]$ cat /etc/httpd/conf.d/software.conf
Alias "/centos7" "/software/centos7"

<Directory /software/centos7>
Options +Indexes

Order allow,deny
Allow from all
Require all granted
</Directory>
Screenshot showing successful directory listing from client machine of centos7 media
Screenshot showing successful directory listing from client machine of centos7 media

The Alias allows you to point to a directory which is outside of Apaches’ DocumentRoot, (typically set to /var/www/html).  The Directory block, contains two things of note.  First, for testing I have added the “Options +Indexes” so that when I try to connect from a web browser on my client machine, I can confirm that I can see the contents of the repository directory.  The second chunk of config, starting “Order all,deny…” is there so that Apache will allow connections to this none standard location.

One thing I did have to do, that I haven’t stated above is allow HTTP connections through the firewall.

This was accomplished by way of a simple one liner;

[toby@rhc-server ~]$ sudo firewall-cmd --zone=public --add-service=http

Note.  To make this new firewall rule permanent you need to use the “–permanent” firewall-cmd option on the command line.  I added this afterwards once I was happy that everything was working.

Configuring a yum .repo file to access the centralised software repository

This is very similar in the steps taken when I setup the local yum repository.  The only difference this time will be that I’ll give it a more meaningful name and the file location will be a http:// address rather that a file:///.

So here is what I have put together;

[root@rhc-client yum.repos.d]# cat CentOS-lab-Media.repo 
[th_lab_server]
baseurl=http://rhc-server.lab.tobyheywood.com/centos7/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

And then, as the saying goes, the proof is in the pudding;

[root@rhc-client yum.repos.d]# yum repolist
Loaded plugins: fastestmirror, langpacks
Repository 'th_lab_server' is missing name in configuration, using id
th_lab_server                                                                                                        | 3.6 kB  00:00:00     
(1/2): th_lab_server/group_gz                                                                                        | 157 kB  00:00:00     
(2/2): th_lab_server/primary_db                                                                                      | 4.9 MB  00:00:00     
Loading mirror speeds from cached hostfile
repo id                                                            repo name                                                          status
th_lab_server                                                      th_lab_server                                                      8,465
repolist: 8,465

Oops, now it would appear I haven’t added a name parameter in the dot repo file.  Let me correct that…

[root@rhc-client yum.repos.d]# cat CentOS-lab-Media.repo
[th_lab_server]
name="CentOS7 Media on rhc-server.lab.tobyheywood.com"
baseurl=http://rhc-server.lab.tobyheywood.com/centos7/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

And now if I run the command “yum repolist” again, it should return the list of enabled repositories without complaining, oh and the repo name column will also show my desired name for my network enabled repository ( a shorter name may be better);

[root@rhc-client yum.repos.d]# yum repolist
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
repo id                                          repo name                                                              status
th_lab_server                                    "CentOS7 Media on rhc-server.lab.tobyheywood.com"             8,465
repolist: 8,465

And there we have it a working centralised repository, if you don’t have access to Red Hat Satellite server or if you don’t want to install the open source version Spacewalk.

I guess the final test, would be to install a couple of packages;

[root@rhc-client yum.repos.d]# yum install iostat
Loaded plugins: fastestmirror, langpacks
th_lab_server                                                                                                        | 3.6 kB  00:00:00
Loading mirror speeds from cached hostfile
No package iostat available.
Error: Nothing to do
[root@rhc-client yum.repos.d]# yum whatprovides iostat
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
th_lab_server/filelists_db                                                                                           | 5.8 MB  00:00:00
sysstat-10.1.5-4.el7.x86_64 : Collection of performance monitoring tools for Linux
Repo        : th_lab_server
Matched from:
Filename    : /usr/bin/iostat

sysstat-10.1.5-4.el7.x86_64 : Collection of performance monitoring tools for Linux
Repo        : @anaconda
Matched from:
Filename    : /usr/bin/iostat

[root@rhc-client yum.repos.d]# yum install sysstat -y
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Package sysstat-10.1.5-4.el7.x86_64 already installed and latest version
Nothing to do

Ah, I guess a demo is never meant to go really smoothly, but this is probably better as it demonstrates some awesome functionality that yum has.

Line 3, shows that it has found my networked repo, line 5 shows that there is no such package in the repo, line 7 highlights a way to find the right package for the command you want to run, lines 10 through 14 show conclusively that it has connected across the network to the repo, and has found a suitable package called sysstat. In line 21, I’m trying to install the package only to be told in line 24 that it’s already installed.

The really keen eyed of you may have also spotted the @anaconda repo, this should have rung an alarm bell in my head to say, hey!  What are you doing?  Its already installed!

Useful link – RPM DB Recovery

Posted on Leave a commentPosted in CentOS 7, Fedora, Linux, RHEL 6, RHEL 7, RPM, System Administration

Every now and then, we find ourselves in a bit of a predicament.  In this instance whilst performing an upgrade on a server, things just weren’t going well and it appeared we had some corruption in the RPM database on one of our servers.

We were seeing segmentation faults when trying to use “rpm”.

The following page on the rpm.org website proved very useful, in getting things back up and running swiftly;

http://www.rpm.org/wiki/Docs/RpmRecovery