Wednesday, 1 May 2013

Linux: Jpeg Image Optimization / Compress Command

I know how to optimize and compress png images using optipng command. line tool. I have lots of images in JPEG format. How do I compress and optimize JPEG images on my amazon cloud account so that I can save bandwidth on cloudfront cdn account? How do I use an image compressor to create lossless compression on JPEG files, with no effect on image quality in bulk using Linux?

JPG file format is recommended for high resolution photographic-style images. You need to use jpegoptim command. It is used to optimize/compress jpeg files. Program supports lossless optimization, which is based on optimizing the Huffman tables. And so called "lossy" optimization where in addition to optimizing Huffman tables user can specify upperlimit for image quality.

Installation

Type the following command:
$ apt-get install jpegoptim
Sample outputs:

[sudo] password for vivek:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libavutil-extra-51 libggiwmh0-target-x libggi2 libgii1 libvo-aacenc0
  libgii1-target-x mplayer-skin-blue libggiwmh0 libggi-target-x
  libvo-amrwbenc0
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  jpegoptim
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 14.0 kB of archives.
After this operation, 77.8 kB of additional disk space will be used.
Get:1 http://mirror.anl.gov/debian/ squeeze/main jpegoptim amd64 1.2.3-2+b1 [14.0 kB]
Fetched 14.0 kB in 1s (11.2 kB/s)
Selecting previously deselected package jpegoptim.
(Reading database ... 333683 files and directories currently installed.)
Unpacking jpegoptim (from .../jpegoptim_1.2.3-2+b1_amd64.deb) ...
Processing triggers for man-db ...
Setting up jpegoptim (1.2.3-2+b1) ...

Examples

The syntax is:

jpegoptim file.jpeg
jpegoptim [options] file.jpeg

Type the following command to optimize photo.jpeg, enter:
$ jpegoptim photo.jpeg
Sample outputs:

photo.jpeg 1312x948 24bit JFIF  [OK] 25226 --> 10744 bytes (57.41%), optimized.

How do I process files in batch?

Use the bash for loop as follows:

 
for i in one.jpeg two.jpeg foo.jpeg; do jpegoptim "$i"; done

 ## process all *.jpeg in the current directory 
for i in *.jpeg; do jpegoptim "$i"; done

Options

From the man page:

-d, --dest=
Sets alternative destination directory where to save optimized files (default is to overwrite the originals). Please note that unchanged
files won't be added to the destination directory. This means if the source file can't be compressed, no file will be created in the destina‐
tion path.
-f, --force
Force optimization, even if the result would be larger than the original file.
-h, --help
Displays short usage information and exits.
-m[0..100], --max=[0..100]
Sets the maximum image quality factor (disables lossless optimization mode, which is by default enabled). This option will reduce quality of
those source files that were saved using higher quality setting. While files that already have lower quality setting will be compressed
using the lossless optimization method.
-n, --noaction
Don't really optimize files, just print results.
-o, --overwrite
Overwrite target file even if it exists (when using -d option).
-p, --preserve
Preserve file modification times.
-q, --quiet
Quiet mode.
-t, --totals
Print totals after processing all files.
-v, --verbose
Enables verbose mode (positively chatty).
--strip-all
Strip all (Comment & Exif) markers from output file. (NOTE! by default only Comment & Exif markers are kept, everything else is discarded)
--strip-com
Strip Comment (COM) markers from output file.
--strip-exif
Strip EXIF markers from output file.
--strip-iptc
Strip IPTC markers from output file.
--strip-icc
Strip ICC profiles from output file.

REFERENCES

jpegoptim home page.
man page - jpegoptim

Linux / Unix Command To Optimize and Compress PNG Files In Bulk

have lots of images in png format (over 250k+ png). Properly formatting and compressing png images can save many bytes of data. How do I compress and optimize png images on my amazon cloud account so that I can save bandwidth on cloudfront cdn account? How do I use an image compressor to create lossless compression on PNG files, with no effect on image quality in bulk using Red Hat Enterprise Linux bash shell?

The following image format is recommended and used by big web sites such as Google / Yahoo / Amazon / Facebook etc:

PNG file format is recommended for web.
GIF file format is recommended for small size images.
JPG file format is recommended for high resolution photographic-style images.
Do not use BMPs or TIFFs.

Say hello to optipng

You need to use a tool called optipng. It is a PNG optimizer that recompresses image files to a smaller size, without losing any information. This program also converts external formats (BMP, GIF, PNM and TIFF) to optimized PNG, and performs PNG integrity checks and corrections. This tool can be installed on any server powered by Unix or Linux operating systems.

Installation

First, turn on EPEL repo and type the following yum command to install optipng:
# yum install optipng
Sample outputs:

 
Loaded plugins: auto-update-debuginfo, protectbase, rhnplugin
0 packages excluded due to repository protections
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package optipng.x86_64 0:0.6.4-1.el6 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
===============================================================================
 Package           Arch             Version               Repository      Size
===============================================================================
Installing:
 optipng           x86_64           0.6.4-1.el6           epel            82 k
 
Transaction Summary
===============================================================================
Install       1 Package(s)
 
Total download size: 82 k
Installed size: 181 k
Is this ok [y/N]: y
Downloading Packages:
optipng-0.6.4-1.el6.x86_64.rpm                          |  82 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : optipng-0.6.4-1.el6.x86_64                                  1/1
  Verifying  : optipng-0.6.4-1.el6.x86_64                                  1/1
 
Installed:
  optipng.x86_64 0:0.6.4-1.el6
 
Complete!

How do I use optipng command?

The syntax is:

 
optipng file
optipng [options] file
optipng [options] input.png

My sample test.png image:

test.png - Original file

To display png image size, type, and compression info, enter:
$ pnginfo -t test.png
Sample outputs:

test.png (tiffinfo compatible labels)...
  Image Width: 1164 Image Length: 911
  Bits/Sample: 8
  Samples/Pixel: 3
  Pixel Depth: 24
  Colour Type (Photometric Interpretation): RGB
  Image filter: Single row per byte filter
  Interlacing: No interlacing
  Compression Scheme: Deflate method 8, 32k window
  Resolution: 0, 0 (unit unknown)
  FillOrder: msb-to-lsb
  Byte Order: Network (Big Endian)
  Number of text strings: 1 of 9
    Software (xTXt deflate compressed): Shutter

Use ls command to see file size:
$ ls -lh test.png
Sample outputs:

-rw-r--r-- 1 vivek vivek 279K Nov 29 00:10 test.png

Use optipng optimization program as follows:

$ cp test.{png,bak}
$ optipng test.png

Sample outputs:

 
OptiPNG 0.6.4: Advanced PNG optimizer.
Copyright (C) 2001-2010 Cosmin Truta.
 
** Processing: test.png
1164x911 pixels, 3x8 bits/pixel, RGB
Input IDAT size = 284778 bytes
Input file size = 285286 bytes
 
Trying:
  zc = 9  zm = 8  zs = 0  f = 0  IDAT size = 215274
 
Selecting parameters:
  zc = 9  zm = 8  zs = 0  f = 0  IDAT size = 215274
 
Output IDAT size = 215274 bytes (69504 bytes decrease)
Output file size = 215374 bytes (69912 bytes = 24.51% decrease)

File size is reduced by 24.51%:
$ ls -lh test.png
Sample outputs:
-rw-r--r-- 1 vivek vivek 211K Nov 29 02:40 test.png
You can verify the quality of both images (click to enlarge):

: test.png – Original file

: test.png – Optimized file

Optimized test.png file ensures that it loads faster for users on slow connections. It will also save 24.51% bytes per image view in terms of both network bandwidth and storage. According to Google:

You should see a benefit for any image file that can be reduced by 25 bytes or more (less than this will not result in any appreciable performance gain).

How do I optimize files in bulk?

Use a bash for loop as follows:

#!/bin/bash
cd /path/to/png/storage/2010/01/c/
mkdir optimized
## store optimized images in optimized directory ##
## Keep file system permission and make a backup of original PNG (see options below)  ##
for i in *.png; do optipng -o5 -quiet -keep -preserve -dir optimized -log optipng.log "$i"; done

You need to make changes to your code. For example, url path to optimize url will be as follows:
http://s0.cyberciti.org/foo/bar/optimized/test.png
Alternatively, you can overwrite existing PNG images:

#!/bin/bash
cd /path/to/png/storage/2010/01/c/
## Overwrite images ##
## Keep file system permission and make a backup of original PNG (see options below)  ##
for i in *.png; do optipng -o5 -quiet -keep -preserve -log optipng.log "$i"; done

Next, purge all images from your CDN account. See how to purge old images / invalidating object from cloud front for more information.

Options

 
General options:
    -fix  enable error recovery
    -force  enforce writing of a new output file
    -keep  keep a backup of the modified files
    -preserve  preserve file attributes if possible
    -quiet  quiet mode
    -simulate  simulation mode
    -snip  cut one image out of multi-image or animation files
    -out <file>  write output file to <file>
    -dir <directory> write output file(s) to <directory>
    -log <file>  log messages to <file>
    --   stop option switch parsing
Optimization options:
    -f  <filters> PNG delta filters (0-5)   default 0,5
    -i  <type>  PNG interlace type (0-1)  default <input>
    -zc <levels> zlib compression levels (1-9)  default 9
    -zm <levels> zlib memory levels (1-9)  default 8
    -zs <strategies> zlib compression strategies (0-3) default 0-3
    -zw <window size> zlib window size (32k,16k,8k,4k,2k,1k,512,256)
    -full  produce a full report on IDAT (might reduce speed)
    -nb   no bit depth reduction
    -nc   no color type reduction
    -np   no palette reduction
    -nx   no reductions
    -nz   no IDAT recoding
Optimization details:
    The optimization level presets
        -o0  <=>  -o1 -nx -nz
        -o1  <=>  [use the libpng heuristics] (1 trial)
        -o2  <=>  -zc9 -zm8 -zs0-3 -f0,5 (8 trials)
        -o3  <=>  -zc9 -zm8-9 -zs0-3 -f0,5 (16 trials)
        -o4  <=>  -zc9 -zm8 -zs0-3 -f0-5 (24 trials)
        -o5  <=>  -zc9 -zm8-9 -zs0-3 -f0-5 (48 trials)
        -o6  <=>  -zc1-9 -zm8 -zs0-3 -f0-5 (120 trials)
        -o7  <=>  -zc1-9 -zm8-9 -zs0-3 -f0-5 (240 trials)
    The libpng heuristics
        -o1  <=>  -zc9 -zm8 -zs0 -f0  (if PLTE is present)
        -o1  <=>  -zc9 -zm8 -zs1 -f5  (if PLTE is not present)
    The most exhaustive search (not generally recommended)
      [no preset] -zc1-9 -zm1-9 -zs0-3 -f0-5 (1080 trials)
Examples:
    optipng file.png    (default speed)
    optipng -o5 file.png   (moderately slow)
    optipng -o7 file.png   (very slow)
    optipng -i1 -o7 -v -full -sim experiment.png

Other png optimization tools for Linux / Unix

Debian / Ubuntu Linux and other operating *nix based systems can use the following tools for the same purpose:

pngcrush - optimizes PNG (Portable Network Graphics) files.
pngnq - tool for optimizing PNG (Portable Network Graphics) images. It is a tool for quantizing PNG images in RGBA format.
pngquant - PNG (Portable Network Graphics) image optimising utility. It is a command-line utility for converting 24/32-bit PNG images to paletted (8-bit) PNGs.

Linux: Force Close A Socket / Port On Server In a TIME_WAIT State

I run a BT client and few other server program on Linux. Sometime these programs get overloaded with too many connections and crashes. If I restart my apps, I see lots of old IPs in a TIME_WAIT state. How do I force and and close everything in a TIME_WAIT state under Linux operating systems?
TIME-WAIT state can exists on either server or client program. It represents waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request.

The/proc/sys/net/ipv4/tcp_fin_timeout setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. This is known as TIME_WAIT state. TIME_WAIT is a normal part of the TCP connection. However, if you must close a socket in TIME_WAIT state, try:

[a] Restart the networking service

[b] cutter command

Finding out current TIME_WAIT settings

Type the following command
$ cat /proc/sys/net/ipv4/tcp_fin_timeout
Sample outputs:

You can lower the value by typing the following command as root user:
# echo 20 > /proc/sys/net/ipv4/tcp_fin_timeout
To set /proc/sys/net/ipv4/tcp_fin_timeout to 20 permanently, edit the file/etc/sysctl.conf and set it as follows:

net.ipv4.tcp_fin_timeout=20

Restating the network service

To restart the network service under RHEL / CentOS based systems, enter:
# service network restart
OR
# /etc/init.d/network restart
Ubuntu / Debian Linux, user try the following command:
$ sudo service networking restart
OR
$ sudo /etc/init.d/networking restart

cutter command

Cutter is an open source program that allows Linux firewall administrators to abort TCP/IP connections routed over the firewall or router on which it is run.

Examples

To cut all connections from 192.168.1.10 to server, enter:
# cutter 192.168.1.5
To cut all ssh connection from 192.168.1.1 to server, type:
# cutter 192.168.1.5 22
To cut all http connection from 192.168.1.5 to ssh server 202.54.1.20, run:
# cutter 202.54.1.20 192.168.1.5 80
See below for 'how to install and use the cutter command' for more information here.

Linux Cutting the tcp/ip network connection with cutter command

by NIXCRAFT on DECEMBER 9, 2005 · 14 COMMENTS· LAST UPDATED SEPTEMBER 29, 2007

Recently I came across very powerful and nifty tool called cutter. Just imagine that people in your private network using peer to peer (P2P) software such as Kazaa, iMesh or others and you want to cut them or just want to cut all ftp connection over your firewall but not all traffic to host. Network security administrators sometimes need to be able to abort TCP/IP connections routed over their firewalls on demand

cutter utility

In the following sample network diagram client workstation 192.168.1.1 sending ftp, http, ssh traffic using 192.168.1.254 (Linux based) router to server outside our network, and you would like to cut ftp traffic without interrupting other connection? So how do you block and cut traffic? Simply, use cutter utility.

client ->    Linux firewall -> Internet --> Servers
FTP    ->    192.168.1.254  -> Internet --> FTP Server
HTTP   ->    192.168.1.254  -> Internet --> HTTP Server
SSH    ->    192.168.1.254  -> Internet --> SSH Server
192.168.1.1

Cutter is an open source program that allows Linux firewall administrators to abort TCP/IP connections routed over Linux based firewall. This tool is very handy in situation like:

To terminate connection such as SSH tunnels or VPNs left by your own users
To abort crackers attacks as soon as they detected
To kill high bandwidth consuming connection
To kill peer-to-peer traffic etc

How do I use cutter command?

Use apt-get to install cutter on a Debian / Ubuntu Linux firewall:
# apt-get install cutter

1) Login to your iptables based firewall router

2) Identify your internal connection (use netstat or tcpdump)

3) Use cutter the command as follows:
cutter {IP-address} {Port}

Examples:
Cut all connections from 192.168.1.5 to server
# cutter 192.168.1.5

Cut all ssh connection from 192.168.1.5 to server
# cutter 192.168.1.5 22

Cut all ssh connection from 192.168.1.5 to ssh server 202.54.1.20
# cutter 202.54.1.20 192.168.1.5 22

Please note that cutter has been designed for use as a administrators tool for Linux firewalls do not use this tool for malicious purpose. For more information about this tool & how actually it works by sending FIN -> ACK -> RST sequence of packets to terminate connection, see theofficial web site.

Update: As pointed out by Mina Naguib you can also use tcpkill command for same purpose.

SEE ALSO

See the following man pages:

man 7 socket
man 7 tcp
man 7 ip
man 5 proc

Linux / Unix: Use rsync Command Over FTP

How do I use rsync command to sync files to a remote host called web-42.vps running
VSFTPD ftp server?
Can you give me rsync ftp example for Unix like operating systems?

Short answer - You can't. rsync can't use ftp as a remote host. So rsync does not work over ftp session/protocol. It only works with rsync protocol.

Long answer - rsync is a tool for doing syncs of two directories on a local system or a combination of local and remote systems. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use. The remote system must have rsync or rsync daemon installed. So, the soultion is below:

The solution

If you must mirror files from local Unix/Linux system to the remote server. Try lftp command. It has builtin mirror which can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server. Mirror can also synchronize directories between two remote servers, using FXP if available. The syntax is:

mirror source target

VIM: Jump Back To Previous or Last Cursor Position

Writing shell scripts in vim I usually go search for functions or something.
Copy lines and then go back to where I was and modify the code.
However, I need to manually navigate way back to the previous / last position.
How can I automatically go back to where I was when I started my search operation under vim?

Vim is a text editor that is upwards compatible to Vi.

It supports the following markers and moving about keys:

[a] '. : Jump to last modification line.

[b] `. : Jump to exact spot in last modification line

[c] CTRL-O : Retrace your movements in file in backwards.

[d] CTRL-I : Retrace your movements in file in forwards.
From the vim help files:

Jumps are remembered in a jump list. With the CTRL-O and CTRL-I command you can go to cursor positions before older jumps, and back again. Thus you can move up and down the list. There is a separate jump list for each window. The maximum number of entries is fixed at 100.

Example

Open a text file called functions.bash, run:
$ vim functions.bash
Move cursor or go to line # 300 by typing the following vim command
300 shift g
Search for a shell function called _addNfsClientUser
/_addNfsClientUser
To jump back to line # 300 or previous position press CTRL-O (press and hold Ctrl key and press letter O). To jump back forwards press CTRL-I (press and hold Ctrl key and press letter I).

HowTo: Check If a Directory Exists In a Shell Script

How do I check if a directory exists in a shell script under Linux or Unix like operating systems?

To check if a directory exists and is a directory use the following syntax:

[ -d "/path/to/dir" ] && echo "Directory /path/to/dir exits." || echo "Error: Directory /path/to/dir does not exits."

The following version also check for symbolic link:
[ -d "/path/to/dir" && ! -L "/path/to/dir" ] && echo "Directory /path/to/dir exits." || echo "Error: Directory /path/to/dir exits but point to $(readlink -f /path/to/dir)."

[ -d "/path/to/dir" && ! -h "/path/to/dir" ] && echo "Directory /path/to/dir exits." || echo "Error: Directory /path/to/dir exits but point to $(readlink -f /path/to/dir)."

Finally, you can use the traditional if..else..fi:

if [ -d "/path/to/dir" ]
then
    echo "Directory /path/to/dir exits."
else
    echo "Error: Directory /path/to/dir does not exits."
fi

Shell script examples to see if a ${directory} exists or not

 
#!/bin/bash
dir="$1"
 
[ $# -eq 0 ] && { echo "Usage: $0 dir-name"; exit 1; }
 
if [ -d "$dir" -a ! -h "$dir" ]
then
   echo "$dir found and setting up new Apache/Lighttpd/Nginx jail, please wait..."
   # __WWWJailSetup "cyberciti.biz" "setup"
else
   echo "Error: $dir not found or is symlink to $(readlink -f ${dir})."
fi

In this example, create directories if does not exits:

# Purpose: Setup jail and copy files
# Category : Core
# Override : No
# Parameter(s) : d => domain name
#                action => setup or update
__WWWJailSetup(){
        local d="$1"
        local action="${2:setup}"       # setup or update???
        local index="<html><head><title>$d</title></head><body><h1>$d</h1></body></html>" # default index.html
        local J="$(_getJailRoot $d)/$d" # our sweet home 
        local _i=""
 
        [ "$action" == "setup" ] && echo "* Init jail config at $J..." || echo "* Updating jail init config at $J..."
        __init_domain_config "$d"
 
        [ "$action" == "setup" ] && echo "* Setting up jail at $J..." || echo "* Updating jail at $J..."
        [ ! -d "$J" ] &&  $_mkdir -p "$J"
 
        for _i in $J/{etc,tmp,usr,var,home,dev,bin,lib64}
        do
                [ ! -d "$_i" ] &&  $_mkdir -p "$_i"
        done
        for _i in $_lighttpd_webalizer_base/$d/stats/{dump,out}
        do
                [ ! -d "$_i" ] &&  $_mkdir -p "$_i"
        done
        for _i in $_lighttpd_webalizer_prepost_base/$d/{pre.d,post.d}
        do
                [ ! -d "$_i" ] &&  $_mkdir -p "$_i"
        done
## truncated 
}

Linux Learning

Wednesday, 1 May 2013

Linux: Jpeg Image Optimization / Compress Command

Installation

Examples

How do I process files in batch?

Options

REFERENCES

Linux / Unix Command To Optimize and Compress PNG Files In Bulk

Say hello to optipng

Installation

How do I use optipng command?

How do I optimize files in bulk?

Options

Other png optimization tools for Linux / Unix

RECOMMENDED READINGS:

Linux: Force Close A Socket / Port On Server In a TIME_WAIT State

Finding out current TIME_WAIT settings

Restating the network service

cutter command

Examples

Linux Cutting the tcp/ip network connection with cutter command

cutter utility

How do I use cutter command?

Linux / Unix: Use rsync Command Over FTP

The solution

VIM: Jump Back To Previous or Last Cursor Position

Example

HowTo: Check If a Directory Exists In a Shell Script

Shell script examples to see if a ${directory} exists or not