Centos Syslog Forward to Splunk

# ### begin forwarding rule ###
# The statement between the begin … end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g., port optional
*.* @@splunk:514

Extending an LVM volume: Physical volumes (partitions) -> Volume groups -> Logical volume -> Filesystem

Logical Volume Management (AKA LVM) is for managing storage space.
In Centos, instead of installing the root filesystem directly to a fixed size partition, we setup LVM by default, and install the root filesystem to a Logical Volume, which may later be expanded, even across multiple physical devices.

How LVM works
In LVM, there are several layers, each builds on top of the other:

PV[s] (Physical Volumes) -> VG[s] (Volume Groups) -> LV[s] (Logical Volumes) -> Filesystems.

Logical Volumes are allocated/extended within the boundaries of their underlying storage pool which is called a Volume Group in LVM terminology.

For example, in Centos the filesystem is installed by default to the /dev/vg_prd00/lv_root Logical Volume, which is allocated within the vg_prd00 Volume Group:

— Logical volume —
LV Name /dev/vg_prd00/lv_root
VG Name vg_prd00
LV UUID —-uuid—-
LV Write Access read/write
LV Status available
# open 1
LV Size 50 GiB
Current LE 12800
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:0
Out of the box the vg_prd00 Volume Group might not have enough free space for you:

# vgdisplay
— Volume group —
VG Name vg_prd00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 63.51 GiB
PE Size 4.00 MiB
Total PE 16258
Alloc PE / Size 16258 / 63.51 GiB
Free PE / Size 0 /0
VG UUID —-uuid—-
We can only extend a Logical Volume within the free space of the underlying Volume Group. How much free space we currently have within the Volume Group can be seen in this part of the output:

Free PE / Size 165 / 660.00 MiB
In the above example we only have 660 MB to allocate to LVMs within the vg_prd00 Volume Group. So if we want to extend the root LV we’ll have to first extend the VG backs it up.

Volume Groups group together Physical Volumes. That’s why they’re called Volume Groups. This command will show us which Physical Volumes have been registered into LVM, and to which volume groups they have been assigned:

# pvdisplay
— Physical volume —
PV Name /dev/sda2
VG Name vg_prd00
PV Size 18.15 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 4645
Free PE 165
Allocated PE 4480
PV UUID —-uuid—-
In this example we only have one Physical Volume (the /dev/sda2 partition) in the turnkey Volume Group.

Extending a Logical Volume
Bottom line: if the underlying Volume Group doesn’t have enough free space, to extend the Logical Volumeyou’ll first have to extend the underlying Volume Group by adding another Physical Volume to it.

In VMWare you could either create a new virtual hard disk device to add to the volume group, or extend an existing virtual hard disk device, create a new partition with cfdisk, and add the new partition to the Volume Group:

# example #1: you’ve added to VMWare a new virtual hard disk called /dev/sdb
pvcreate /dev/sdb
vgextend vg_prd00 /dev/sdb

# example #2: you’ve expanded the existing sda hard disk
cfdisk /dev/sda # creating /dev/sda3 (you need to reboot before you can see this)
pvcreate /dev/sda3
vgextend vg_prd00 /dev/sda3
After you’ve extended the Volume Group, you are free to extend the underlying Logical Volume:

# lvextend -L+10G /dev/vg_prd00/root
Extending logical volume root to XXXX GiB
Logical volume root successfully resized
Finally, you’ll have to resize the filesystem within /dev/turnkey/root so it can see that the underlying block device just got 10G bigger:

# resize2fs /dev/vg_prd00/root
resize2fs 1.41.11 (14-Mar-2010)
Filesystem at /dev/vg_prd00/root is mounted on /; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/vg_prd00/root to XXXX (4k) blocks.
The filesystem on /dev/vg_prd00/root is now XXXX blocks longer.

#xfs_growfs /dev/vg_*name*/lv_*name*

Installing VMware Tools on CentOS

Following is the steps I used to solve installing VMware Tools on CentOS
1.In Sphere Client: Guest > Install/update VMware Tools
2. On the VM console in command line, being a server I do not have a GUI installed:
3. mkdir /cdrom, or any other mount point you like
4. mount the vitural cdrom from step 1; mount /dev/cdrom /cdrom
5. cd /tmp
6. tar -xvf /cdrom/VMwareTools (tab to auto-complete)
7. cd vmware (tab to auto-complete)
8. ./vmware-install.pl; accept defaults, you might have to install perl as well
9. umount /cdrom
10. In Vsphere Client: Guest > Install/upVMware Tools > OK
11. To confirm the install; in the client > VM Summary tab and look for OK next to VMWare Tools

Soft lockup messages from Linux kernel running in an SMP-enabled virtual machine


When running a Linux kernel in a symmetric multiprocessing (SMP) enabled virtual machine, messages similar to BUG: soft lockup detected on CPU#1! are written to the message log file. The exact format of these messages vary from kernel to kernel, and might be accompanied by a kernel stack backtrace.
Many Linux kernels have a soft lockup watchdog thread, and report soft lockup messages if that watchdog thread does not get scheduled for more than 10 seconds. On a physical host, a soft lockup message generally indicates a kernel bug or hardware bug. When running in a virtual machine, this might instead indicate high levels of overcommitment (especially memory overcommitment) or other virtualization overheads.


The soft lockup messages are not kernel panics, and can be safely ignored.Some kernels allow you to adjust the soft lockup threshold by running the command:echo time > /proc/sys/kernel/softlockup_thresh

Where time is the number of seconds after which a soft lockup is reported. The default is generally 10 seconds.

What the Error Looks Like:

abrt_version:   2.0.8


cmdline:        /usr/bin/ksh ./update_archive.ksh RMUAT

executable:     /bin/ksh93

kernel:         2.6.39-400.212.1.el6uek.x86_64

last_occurrence: 1403155682

pid:            25607

pwd:            /oracle/scripts/update_archive

time:           Thu 19 Jun 2014 12:28:02 AM CDT

uid:            502

username:       oracle


sosreport.tar.xz: Binary file, 3959288 bytes


















:Limit                     Soft Limit           Hard Limit           Units

:Max cpu time              unlimited            unlimited            seconds

:Max file size             unlimited            unlimited            bytes

:Max data size             unlimited            unlimited            bytes

:Max stack size            10485760             33554432             bytes

:Max core file size        0                    unlimited            bytes

:Max resident set          unlimited            unlimited            bytes

:Max processes             16384                16384                processes

:Max open files            1024                 65536                files

:Max locked memory         65536                65536                bytes

:Max address space         unlimited            unlimited            bytes

:Max file locks            unlimited            unlimited            locks

:Max pending signals       387052               387052               signals

:Max msgqueue size         819200               819200               bytes

:Max nice priority         0                    0

:Max realtime priority     0                    0

:Max realtime timeout      unlimited            unlimited            us



:00400000-0055b000 r-xp 00000000 fc:00 261718                             /bin/ksh93

:0075a000-0076d000 rw-p 0015a000 fc:00 261718                             /bin/ksh93

:0076d000-00773000 rw-p 00000000 00:00 0

:0096c000-0096e000 rw-p 0016c000 fc:00 261718                             /bin/ksh93

:3e8dc00000-3e8dc20000 r-xp 00000000 fc:00 1177356                        /lib64/ld-2.12.so

:3e8de1f000-3e8de20000 r–p 0001f000 fc:00 1177356                        /lib64/ld-2.12.so

:3e8de20000-3e8de21000 rw-p 00020000 fc:00 1177356                        /lib64/ld-2.12.so

:3e8de21000-3e8de22000 rw-p 00000000 00:00 0

:3e8e000000-3e8e002000 r-xp 00000000 fc:00 1177390                        /lib64/libdl-2.12.so

:3e8e002000-3e8e202000 —p 00002000 fc:00 1177390                        /lib64/libdl-2.12.so

:3e8e202000-3e8e203000 r–p 00002000 fc:00 1177390                        /lib64/libdl-2.12.so

:3e8e203000-3e8e204000 rw-p 00003000 fc:00 1177390                        /lib64/libdl-2.12.so

:3e8e400000-3e8e58b000 r-xp 00000000 fc:00 1177360                        /lib64/libc-2.12.so

:3e8e58b000-3e8e78a000 —p 0018b000 fc:00 1177360                        /lib64/libc-2.12.so

:3e8e78a000-3e8e78e000 r–p 0018a000 fc:00 1177360                        /lib64/libc-2.12.so

:3e8e78e000-3e8e78f000 rw-p 0018e000 fc:00 1177360                        /lib64/libc-2.12.so

:3e8e78f000-3e8e794000 rw-p 00000000 00:00 0

:3e8f000000-3e8f083000 r-xp 00000000 fc:00 1177398                        /lib64/libm-2.12.so

:3e8f083000-3e8f282000 —p 00083000 fc:00 1177398                        /lib64/libm-2.12.so

:3e8f282000-3e8f283000 r–p 00082000 fc:00 1177398                        /lib64/libm-2.12.so

:3e8f283000-3e8f284000 rw-p 00083000 fc:00 1177398                        /lib64/libm-2.12.so

:3e98400000-3e98402000 r-xp 00000000 fc:00 1177423                        /lib64/libutil-2.12.so

:3e98402000-3e98601000 —p 00002000 fc:00 1177423                        /lib64/libutil-2.12.so

:3e98601000-3e98602000 r–p 00001000 fc:00 1177423                        /lib64/libutil-2.12.so

:3e98602000-3e98603000 rw-p 00002000 fc:00 1177423                        /lib64/libutil-2.12.so

:7fea21b5d000-7fea21bcd000 rw-p 00000000 00:00 0

:7fea21bcd000-7fea27a5e000 r–p 00000000 fc:00 1441433                    /usr/lib/locale/locale-archive

:7fea27a5e000-7fea27aaa000 rw-p 00000000 00:00 0

:7fea27ab8000-7fea27ab9000 rw-p 00000000 00:00 0

:7fffdfc56000-7fffdfc77000 rw-p 00000000 00:00 0                          [stack]

:7fffdfde7000-7fffdfde8000 r-xp 00000000 00:00 0                          [vdso]

:ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]




:pos: 0

:flags:     00


:pos: 199

:flags:     0100001


:pos: 199

:flags:     0100001

Non-Root User Permissions Oracle Linux

I’m working on a system recently migrated to Oracle Linux 6.6 from a very old Solaris system. There is a CIFS mount from a Windows 2012r2 server that existed on the old system. The raw mount point has 777 directory permissions.

[root@localhost ~]# ls -ld /datastore/
drwxrwxrwx 2 root root 4096 Jan 6 09:50 /datastore/
When the mount is active the permissions are:

[root@localhost ~]# ls -ld /datastore/
drwxr-xr-x 1 root root 634564 Jan 6 09:50 /datastore/
Users other than root cannot write to the share or create files. Looking at the old server, the permissions on files and subdirectories within the same share have the setuid bit. This is not present on the new system. The /etc/fstab looks like:

//cifshost/datastore /datastore cifs username=user,password=password,domain=mydomain.local 0 0

You’ll need to change /etc/fstab and add the file_mode=0666,dir_mode=0777 mount options.

//cifshost/share/datastore /datastore cifs user=user,pass=password,file_mode=0666,dir_mode=0777 0 0

And you should be good to go!

Vmware Tools on CentOS 6.x or 7 x64


Running Vsphere 5.x and installed the VMWare tools on a CentOS 6.x x64 system.  After an OS update that requires reboot vmware-tools does not automatically startup.

As of version (build-1015158) the vmware tools install script vmware-install.pl does not create a start script in /etc/init.d since it now uses upstart.

Additional information: When you installed vmware-tools you did not Enable automatic building and installation of kernel modules at boot.


1.) You can re-run /usr/bin/vmware-config-tools.pl after each OS update that modifies the kernel.2.) You can run /usr/bin/vmware-config-tools.pl and ENABLE the automatic building and installation of kernel modules at boot (Note: you will need to remember to enable this option with subsequent vmware-tools upgrades)

3.) You can create your own init script from the services.sh script located in the /etc/vmware-tools directory.

From command line:
[root@host]# cp /etc/vmware-tools/services.sh /etc/init.d/vmware-tools
[root@host]# vim /etc/init.d/vmware-tools

Paste the following just below the line ##VMWARE_INIT_INFO## and save:

# chkconfig: 235 03 99

[root@host]# chkconfig –add vmware-tools (two dashes)
[root@host]# chkconfig vmware-tools on

Verify that it works:
[root@host]# service vmware-tools restart

Unix and Active Directory Integration

Step by step,

yum install nscd samba samba-common samba-client samba-winbind

yum upgrade

vi /etc/hosts dc.domain.com pdc01

Run authconfig-tui
Authentication Configuration – check Cache Information, Use Winbind, Use MD5 Passwords, Use Shadow Passwords, Use Winbind Authentication
Winbind Settings – check ads, type the short name of the domain, example.com needs to be just example in this field, enter FQDN of domain controllers, ADS realm is FQDN of primary DC, check /bin/bash
Click on Join Domain
Enter credentials for a domain administrator and make sure the server successfully joined the domain

vi /usr/local/bin/bash-wrapper


# This script restricts shell access to privileged users. The “template shell”
# option in the ‘/etc/samba/smb.conf’ file should be set to call this wrapper.

# Get group memberships for this user.

# Grant shell access to users that are in the local wheel group.
if /bin/echo “$BFN_ID” | /bin/grep -P ‘[=,][0-9]{1,8}(wheel)’ > /dev/null
exec /bin/bash –login “$@”

# Grant shell access to users that are in the domain administrators group.
if /bin/echo “$BFN_ID” | /bin/grep -P ‘[=,][0-9]{1,8}(domain admins)’ > /dev/null
exec /bin/bash –login “$@”

# Else print a notice and just exit.
echo “Shell access to this computer is disabled.”

# eof

vi /usr/local/bin/ad-phase2.sh

# ad-phase2.sh – Phase 2
# Description: This script automates the process of joining a linux box
# to an AD domain. The process is divided in two parts.
# Please edit the relevant parts of the script below prior running it

# This block doesn’t need to be edited
#sed -i ‘s%protocols: files%protocols: files winbind%g’ /etc/nsswitch.conf
#sed -i ‘s%rpc: files%rpc: files winbind%g’ /etc/nsswitch.conf
#sed -i ‘s%netgroup: files%netgroup: files winbind%g’ /etc/nsswitch.conf
#sed -i ‘s%automount: files%automount: files winbind%g’ /etc/nsswitch.conf

# The following line allows users to logon without the ugly EXAMPLEuser syntax
sed -i ‘s%winbind use default domain = false%winbind use default domain = true%g’ /etc/samba/smb.conf

# More parameters to make life easier with UID and GID correspondances
sed -i ‘s% template shell = /bin/bash% template shell = /usr/local/bin/bash-wrapper%g’ /etc/samba/smb.conf
sed -i ‘/ winbind offline logon = false/a winbind enum users = true’ /etc/samba/smb.conf
sed -i ‘/winbind enum users = true/a winbind enum groups = true’ /etc/samba/smb.conf
sed -i ‘/winbind enum groups = true/a winbind cache time = 5’ /etc/samba/smb.conf
sed -i ‘/winbind cache time = 5/a winbind nested groups = true’ /etc/samba/smb.conf

# This line will allow for home folders to be created in /home/DOMAIN/username upon first login
echo “session optional pam_mkhomedir.so skel=/etc/skel/ umask=0022” >> /etc/pam.d/system-auth

# The following line will allow all the users within the Domain Admins group to sudo on the server
echo “%domain admins ALL=(ALL) ALL” >> /etc/sudoers

# Replace “base OU=Users,DC=example,DC=com” with the container of the users you want to allow on the box
sed -i ‘s%base dc=*******,dc=com%base OU=Users,DC=*******************,DC=com%g’ /etc/openldap/ldap.conf
chmod +x /usr/local/bin/ad-phase2.sh
chmod +x /usr/local/bin/bash-wrapper
service winbind restart
service nscd restart

Reboot for good measure.