kernel: BUG: soft lockup – CPU#0 stuck for 67s!

Posted on Leave a commentPosted in Kernel Tuning, Linux, RHEL 6, System Administration

Over the weekend I was confronted by the above error being repeated on the console of a VM running Oracle RDBMS.

This error occurs when there is a shortage of CPU resources.  For me the solution was a quick shut down of the VM and increasing the available CPU resources.  However there are more ways to skin this cat…

There is also a kernel parameter which can be tweaked;

kernel.softlockup_thresh=x

Were “x” is the threshold (in seconds) you want to allow the kernel to wait before decided there has been a soft lockup.

The Red Hat documentation showed a threshold of 30 seconds.  So I would recommend a bit of experimentation if you feel that 30 seconds is not high enough.  Or throw more resources at it.

Reference Material

Featured image: The Old Lockup was made available by John Powell on Flickr.

Kernel Tuning: kernel.core_pattern

Posted on Leave a commentPosted in Kernel Tuning, Linux

There are two types of core dump which can take place on a linux based system.

  1. Application core dump
  2. Kernel core dump

kernel.core_pattern tunable relates to the former (application dump).  It is used to control where application core dumps are saved and also how they will be named.

Application core dumps can occur, for example, when there is a memory segmentation fault.

Here is an example of the kernel.core_pattern when set in the sysctl.conf file;

<i>kernel.core_pattern = /you/define/the/path/appcore-%e-%s-%u-%g-%p-%t

The formatting placeholders used above have the following definitions;

# appcore = generic reference to what is in the file
# %e = executable file name (without the path being prefixed
# %s = the number of the signal which caused the application to crash out and dump it's contents
# %u = the real ID which was running the dumped process
# %g = the real GID which was running the dumped process
# %p = the pid of the dumped process
# %t = the time of the dump

The following is stolen from the core(5) man page;

<pre>   <b>Naming of core dump files</b>
       By default, a core dump file is named <i>core</i>, but the
       <i>/proc/sys/kernel/core_pattern file (since Linux 2.6 and 2.4.21) can
       be set to define a template that is used to name core dump files.
       The template can contain % specifiers which are substituted by the
       following values when a core file is created:

           %%  a single % character
           %c  core file size soft resource limit of crashing process (since
               Linux 2.6.24)
           %d  dump mode—same as value returned by <a href="http://man7.org/linux/man-pages/man2/prctl.2.html">prctl(2)</a> <b>PR_GET_DUMPABLE</b>
               (since Linux 3.7)
           %e  executable filename (without path prefix)
           %E  pathname of executable, with slashes ('/') replaced by
               exclamation marks ('!') (since Linux 3.0).
           %g  (numeric) real GID of dumped process
           %h  hostname (same as <i>nodename</i> returned by <a href="http://man7.org/linux/man-pages/man2/uname.2.html">uname(2)</a>)
           %i  TID of thread that triggered core dump, as seen in the PID
               namespace in which the thread resides (since Linux 3.18)
           %I  TID of thread that triggered core dump, as seen in the
               initial PID namespace (since Linux 3.18)
           %p  PID of dumped process, as seen in the PID namespace in which
               the process resides
           %P  PID of dumped process, as seen in the initial PID namespace
               (since Linux 3.12)
           %s  number of signal causing dump
           %t  time of dump, expressed as seconds since the Epoch,
               1970-01-01 00:00:00 +0000 (UTC)
           %u  (numeric) real UID of dumped process

       A single % at the end of the template is dropped from the core
       filename, as is the combination of a % followed by any character
       other than those listed above.  All other characters in the template
       become a literal part of the core filename.  The template may include
       '/' characters, which are interpreted as delimiters for directory
       names.  The maximum size of the resulting core filename is 128 bytes
       (64 bytes in kernels before 2.6.19).  The default value in this file
       is "core".  For backward compatibility, if
       <i>/proc/sys/kernel/core_pattern</i> does not include "%p" and
       <i>/proc/sys/kernel/core_uses_pid</i> (see below) is nonzero, then .PID will
       be appended to the core filename.

       Since version 2.4, Linux has also provided a more primitive method of
       controlling the name of the core dump file.  If the
       <i>/proc/sys/kernel/core_uses_pid</i> file contains the value 0, then a core
       dump file is simply named <i>core</i>.  If this file contains a nonzero
       value, then the core dump file includes the process ID in a name of
       the form <i>core.PID</i>.

       Since Linux 3.6, if <i>/proc/sys/fs/suid_dumpable</i> is set to 2
       ("suidsafe"), the pattern must be either an absolute pathname
       (starting with a leading '/' character) or a pipe, as defined below.

Linux Kernel Tweaks: kernel.core_uses_pid

Posted on Leave a commentPosted in Kernel Tuning, Linux

The sysctl option “kernel.core_uses_pid” comes into play when a vmcore file is being dumped.  It can be set in one of two ways;

kernel.core_uses_pid = 0

or

kernel.core_uses_pid = 1

As you can probably tell this is a binary settings.  It’s either enabled (1) or it’s not (0).

The exact text from the sysctl.txt file is;

==============================================================

core_uses_pid:

The default coredump filename is "core".  By setting
core_uses_pid to 1, the coredump filename becomes core.PID.
If core_pattern does not include "%p" (default does not)
and core_uses_pid is set, then .PID will be appended to
the filename.

==============================================================

For Fedora 22 this parameter is set to “1” by default.

Kernel Tuning: kernel.sched_child_runs_first

Posted on Leave a commentPosted in Kernel Tuning, Linux, RHEL 6

So, to kick start the Linux Kernel Tuning series I appear to have embarked on, first on the list is the kernel tunable parameter – kernel.sched_child_runs_first.

Disclaimer – see bottom of page.

The not so important disclaimer.  Basically, I have limited knowledge of C/C++ and therefore I am relying purely on third party information to provide the below.  So if you spot an inaccuracy let me know.

What is it for?

The idea behind the kernel.sched_child_runs_first parameter is that there may be times where a process (the parent)  will spawn a (number of) child process(es).  In this scenario it is likely that the child process(es) are the ones doing the actual grunt work and that the parent process is potentially just over seeing the running of the child processes.

The purpose of this variable is to set a preference that if at all possible, the child process should be run before the parent.  The thinking is that this could shave some processing time off of the overall total.

(more…)