Scheduling Tasks With Cron Jobs

Scheduling Tasks With Cron Jobs

Linux systems ships with crond daemon which is enabled by default. There are many configuration files for this daemon, one is for user (which is controlled by crontab command) and others are in use by the daemon itself. If the commands run from the cron job will produce output, this output will be emailed to a user.

Normal users can user crontab to manage their jobs. root can use crontab -u $username to manage the cron jobs for other users.

Job Format
Jobs consist of six fields: minutes, hours, day of month, month, day of week, command. For first five fields you can use the same syntax:

field          allowed values
-----          --------------
minute         0-59
hour           0-23
day of month   1-31
month          1-12 (or names, see below)
day of week    0-7 (0 or 7 is Sunday, or use names)

* - means do not care

There are also some extensions “nicknames”, which replace the 5 initial fields:

@reboot    :    Run once after reboot.
@yearly    :    Run once a year, ie.  "0 0 1 1 *".
@annually  :    Run once a year, ie.  "0 0 1 1 *".
@monthly   :    Run once a month, ie. "0 0 1 * *".
@weekly    :    Run once a week, ie.  "0 0 * * 0".
@daily     :    Run once a day, ie.   "0 0 * * *".
@hourly    :    Run once an hour, ie. "0 * * * *".

For example:

0 9 2 3 * /scripts/yearly_backup.sh - will run at 9:00 AM on 3 March
*/5 9-15 * Jul 5 /script.sh - will run each 5 minutes between 9:00-16:00 on every Friday in July

See crontab -h or man 5 crontab for more info.

System cron jobs
System cron jobs are not defined with crontab command. They can be found under /etc/cron.daily/ /etc/cron.hourly/ /etc/cron.monthly/ /etc/cron.weekly/ /etc/cron.d/. These directories contains executable scripts and not cron configuration files.

/etc/cron.hourly/ is executed by run-parts from a job defined in /etc/cron.d/0hourly. Daily, weekly and monthly are executed by run-parts as well, but from a different config file: /etc/anacrontab.

Securing cron daemon
There two files you need to know. /etc/cron.allow and /etc/cron.deny. Using the first one will let you use whitelist. Second one is blacklist. I recommend using whitelist with /etc/cron.allow. Each line write down the username that is allowed to create cron jobs.

Using Arrays in Bash

Using Arrays in Bash

In bash you can use arrays. Lets see some examples.

Create new array:

ARR=("a" "b" "c")

Show all array values:

[root@knesenko-centos7 ~]# echo "${ARR[@]}"
a b c
[root@knesenko-centos7 ~]# echo "${ARR[*]}"
a b c
[root@knesenko-centos7 ~]#

Difference between ‘*’ and ‘@’. Same as in positional parameters:

[root@knesenko-centos7 ~]# cat "${ARR[@]}"
cat: : No such file or directory
cat: b: No such file or directory
cat: c: No such file or directory
cat: c: No such file or directory
[root@knesenko-centos7 ~]# cat "${ARR[*]}"
cat:  b c c: No such file or directory
[root@knesenko-centos7 ~]#

Loop over all array elements:

for i in "${ARR[@]}"; do
    echo "${i}"
done

Show how many values are in array:

[root@knesenko-centos7 ~]# echo "${#ARR[@]}"
3

Access specific value:

[root@knesenko-centos7 ~]# echo "${ARR[1]}"
b

Add value to array:

ARR+=("d")
[root@knesenko-centos7 ~]# echo "${ARR[@]}"
a b c d

Remove a value from array:

[root@knesenko-centos7 ~]# echo "${ARR[@]/a}"
 b c c
[root@knesenko-centos7 ~]# ARR=("${ARR[@]/a}")
[root@knesenko-centos7 ~]# echo "${ARR[@]}"
 b c c
[root@knesenko-centos7 ~]# 

Schedule One Time Tasks Linux

Schedule One Time Tasks Linux

From time to time, you want to run a command or set of commands at a set point of the future. These scheduled commands usually called tasks or jobs. One of the solutions is to use at. This is not a standalone tool, but rather a system daemon atd. atd can be found in the at package.

Scheduling Jobs

Create a new job (to finish the input use CTRL+D):

at 

To show the at queue.lists the user’s pending jobs, unless the user is the superuser; in that case, everybody’s jobs are listed. The format of the output lines (one for each job) is: Job number, date, hour, queue, and username.

[root@knesenko-centos7 ~]# atq
4	Fri Jul 24 09:42:00 2015 a root

Show the job content:

 at -c #job number

Remove the job:

atrm #job number

Time formats:

noon
midnight
teatime
tomorrow
noon tomorrow
next week
next monday
fri
NOV
9:00 AM
1:45 PM
1430
1:45 PM tomorrow
1:45 PM next month
1:45 PM 10/21
1:45 PM Oct 21
1:45 PM October 21 2014
1:45 PM 10/21/2014
1:45 PM October 21 2014
1:45 PM 21.10.14
1:45 PM October 21 2014
now + 30 minutes
now + 1 hour
now + 2 days

So for example you can run:

at now + 30 minutes
at> echo ok
at> 
job 4 at Fri Jul 24 09:42:00 2015

Securing at daemon
There two files you need to know. /etc/at.allow and /etc/at.deny. Using the first one will let you use whitelist. Second one is blacklist. I recommend using whitelist with /etc/at.allow. Each line write down the username that is allowed to create at jobs.

Batch
batch executes commands when system load levels permit; in other words, when the load average drops below 0.8, or the value specified in the invocation of atd.

Automating Installation With Kickstart

Automating Installation With Kickstart

Introduction

A systems administrator can automate the installation using a feature called kickstart. Anaconda is a program that installs the system and it needs to be told how to install the system: network configuration, disks, partitions, users, groups etc. A kickstart installation uses uses a text file to provide all the answers to these questions. By default, the installation is interactive and all the questions should be answered by a user. The recommended approach to creating Kickstart files is to perform a manual installation on one system first. After the installation completes, all choices made during the installation are saved into a file named anaconda-ks.cfg, located in the /root/ directory on the installed system. You can then copy this file, make any changes you need, and use the resulting configuration file in further installations.

Configuration Commands

url: Specifies the location of the installation media

url --url="http://mirror.centos.org/centos/7/os/x86_64"

repo: This option tells anaconda which repository to use and from where to install packages

repo --name="My example repo" --baseurl="http://example.com/Centos/7/packages"

text: Forces text mode installation

vnc: Allows graphical installation via VNC

vnc --password=testpassword

For more options please navigate here

Example of a kickstart file
You can use ksvalidator command to verify kickstart file syntax

#version=RHEL7
# System authorization information
auth --enableshadow --passalgo=sha512

# Use network installation
url --url="http://mirror.centos.org/centos/7/os/x86_64"
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_US.UTF-8

# Network information
network  --bootproto=dhcp --device=eno16777736 --ipv6=auto --activate
network  --hostname=knesenko-centos7
# Root password
rootpw --iscrypted $6$PRh...iI7XKmbA8J1
# System timezone
timezone Asia/Jerusalem --isUtc --nontp
user --groups=wheel --name=knesenko --password=$6$PRh...iI7XKmbA8J1 --iscrypted --gecos="knesenko"
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
# Partition clearing information
clearpart --none --initlabel 
# Disk partitioning information
part /boot --fstype="ext4" --ondisk=sda --size=500
part pv.10 --fstype="lvmpv" --ondisk=sda --size=19979
volgroup centos_knesenko-centos7 --pesize=4096 pv.10
logvol swap  --fstype="swap" --size=2048 --name=swap --vgname=centos_knesenko-centos7
logvol /  --fstype="ext4" --grow --maxsize=51200 --size=1024 --name=root --vgname=centos_knesenko-centos7

%packages
@core

%end

Publish the Kickstart configuration file to Anaconda
The installer must be able to access the kickstart file during the installation. The most common way is through a network server, such as: http, ftp or nfs.

Boot Anaconda and point to Kickstart file
To start a Kickstart installation, a special boot option (inst.ks=) must be specified when booting the system.

inst.ks=/http://example.com/Centos/7/kickstart.cfg

Configure NTP Client and Server Centos

Configure NTP Client and Server Centos

Install NTP

The ntp package is same for client and server. This package will synchronize your server’s time via NTP protocol. In my example server’s address is 192.168.0.1 and client’s address is 192.168.0.2.

# yum -y install ntp

 

Configure NTP server

Make sure the following lines exists in /etc/ntp.conf. Stratum is used to synchronize the time with the server based on distance.

server ntp.server.com
server 127.127.1.0
furge 127.127.1.0 stratum 10

 

Start the NTP service

For SysVinit based systems run:

service ntpd start

For Systemd systems run:

systemctl start ntpd.service

 

Configure NTP client

Make sure the following line exists in /etc/ntp.conf.

It might take about 15 minutes until ntp will synchronize the clock.
server 192.168.0.1

Now start the NTP service same as above.

 

Debug the NTP service

There are several useful commands you can use when working with NTP. Here are some of them.

To sync the clock:

service ntpd stop && ntpd -dgq && service ntpd start

Show ntp sync status:

ntpq -pn

Checking the Memory Usage Linux

Checking Memory in Linux

Sometimes you want to check your memory usage and there are a lot of tools to do so. I”ll try to explain about some most useful tools to check memory usage of your system.

free

knesenko@knesenko-centos7:~$ free -m
total used free shared buffers cached
Mem: 3778 861 2916 16 84 451
-/+ buffers/cache: 325 3452
Swap: 2047 0 2047

What you can see here, is that total amount of your memory is 3778 MB325 MB are used by processes. 3452 MB are free yo use. Last line shows you the swap usage.


/proc/meminfo

Also you can check /proc/meminfo

knesenko@knesenko-centos7:~$ egrep --color 'Mem|Cache|Swap' /proc/meminfo

MemTotal: 3869004 kB

MemFree: 2986092 kB

MemAvailable: 3392864 kB

Cached: 479408 kB

SwapCached: 0 kB

SwapTotal: 2097148 kB

SwapFree: 2097148 kB

vmstat

vmstat shows you a bit more info about your memory and other resources usage.

knesenko@knesenko-centos7:~$ vmstat

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

r b swpd free buff cache si so bi bo in cs us sy id wa st

2 0 0 2986092 87172 462452 0 0 2  8  39 73 0  0  100 0 0

procs: 
r: How many processes are waiting for CPU time.
b: Wait Queue – Process which are waiting for I/O (disk, network, user input,etc..) 

memory: 
swpd: shows how many blocks are swapped out to disk (paged). Total Virtual memory usage. 

free: Idle Memory

buff: Memory used as buffers, like before/after I/O operations
cache: Memory used as cache by the OS

swap: 
si: How many blocks per second the operating system is swapping in. i.e Memory swapped in from the disk (Read from swap area to Memory)
so: How many blocks per second the operating system is swapped Out. i.e Memory swapped to the disk (Written to swap area and cleared from Memory). In Ideal condition, We like to see si and so at 0 most of the time, and we definitely don’t like to see more than 10 blocks per second.

io: 
bi: Blocks received from block device – Read (like a hard disk) 
bo: Blocks sent to a block device – Write

system: 
in: The number of interrupts per second, including the clock. 
cs: The number of context switches per second. 

CPU: 
us: percentage of cpu used for running non-kernel code. (user time, including nice time) 
sy: percentage of cpu used for running kernel code. (system time – network, IO interrupts, etc) 
id: cpu idle time in percentage.
wa: percentage of time spent by cpu for waiting to IO.

knesenko@knesenko-centos7:~$ vmstat -s
      3869516 K total memory
      1758256 K used memory
       279100 K active memory
      1317776 K inactive memory
      2111260 K free memory
        59584 K buffer memory
      1331872 K swap cache
      2097148 K total swap
            0 K used swap
      2097148 K free swap
        15833 non-nice user cpu ticks
            9 nice user cpu ticks
        26954 system cpu ticks
      6886801 idle cpu ticks
          319 IO-wait cpu ticks
            5 IRQ cpu ticks
         2636 softirq cpu ticks
            0 stolen cpu ticks
       340359 pages paged in
      1229646 pages paged out
            0 pages swapped in
            0 pages swapped out
     10782519 interrupts
     21597473 CPU context switches
   1424333862 boot time
         8042 forks

 

Also I would recommend to use top, htop commands.