pcapstreamer – A packet dumper

Hi guys,

Happy Holidays!!

This year is about to finish, thinking about this year, lot of things happened, love, break-up, home, health, mother, father, work and more importently passion. Well, all is well and life is moving ahead. Still I’m travelling alone, in my own path. (What the hell am I, this is suppose to be technical post, shit!! crap philosophy!!)

I got some free time and spent that time learning libpcap. For those who don’t know, it is used in most of the network monitoring/capturing tools in *nix world. Very powerful.

The ‘tcpdump(1)‘ command is one such tool which uses libpcap (actually they are the one who created libpcap from tcpdump) to dump information about packets. It has a robest filtering mechanism to narrow down packet capturing to specific packets.

While trying to understand filter expressions in tcpdump, I got an Idea, I thought why not just convert the bytes in packets to strings and print them in stdout, this way, we can see the exact bytes, so further processing can be done my other unix tools (like awk, perl etc.,).

So, I just wrote a tool called ‘pcapstreamer‘ to capture packets from linux’s ‘any’ psudo-interface. Its very simple tool, you need to run this tool as root user. It just dump packets, thats all. Here is an example, this shows one packet dumped into stdout.

$ sudo ./pcapstreamer
[cl:76 l:76 t:20111226085033.641612] 00000000 00000000 00000011 00000100 00000000 00000110 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00001000 00000000 01000101 00000000 00000000 00111100 01000111 10001100 01000000 00000000 01000000 00000110 11110101 00101101 01111111 00000000 00000000 00000001 01111111 00000000 00000000 00000001 11100101 01100011 00010101 10110011 01111000 00011101 00100110 01010100 00000000 00000000 00000000 00000000 10100000 00000010 10000000 00011000 11111110 00110000 00000000 00000000 00000010 00000100 01000000 00001100 00000100 00000010 00001000 00001010 00000000 10100010 01100001 11011000 00000000 00000000 00000000 00000000 00000001 00000011 00000011 00000101

Here ‘cl:76’ and ‘l:76’ indicates ‘captured length’, ‘t:20111226085033.641612’ indicates ‘timestamp’ in localtime. Other strings are just pure raw packet.

Linux Cooked Header

To understand first 16 bytes, we need to understand ‘Linux Cooked Header‘. First 2 bytes “00000000 00000000’ or “0x00” represents that this is an incoming packet. To understand the next 2 bytes, we need to refer linux’s ARPHRD_. 3rd and 4th bytes “00000011 00000100” or “decimal 772” indicates that this packet is coming into loopback interface. 5th and 6th bytes “00000000 00000110” or “0x0006” indicates the length of link-level address, the next 8 bytes (7th byte to 14th byte) represents the link-level address, however we should take only the next 6 bytes as link-level address, two more bytes (13th and 14th) are padded with zero. 15th and 16th bytes “00000000 00001000” or “0x0008” represents ‘ethertype‘ as ‘ip’, this tells us that this is an ‘ip’ packet. This ends the link-level header (data-link layer in OSI). We are now moving to ‘ip’ header (network layer in OSI)

IP Header

To understand details from 17th byte to 36th byte, we need to refer IP Header. Higher order 4 bits in 17th byte “0100” or “0x4” indicates that this ip packet is an ipv4 packet. Lower order 4 bits in 17th byte “0101” or “0x5” indicates IHL (Internet Header Length) usually this defaults to 5. 18th bytes represents ‘differentiated services’ usually 0. 19th and 20th bytes “00000000 00111100” or “0x003c” or “Decimal 60” represents remaining bytes count (CaptureLength minus Linux-Cooked-Header length). 21st and 22nd bytes “01000111 10001100” indicates identification. Higher order 3 bits in 23rd and 24th bytes “010” indicates that this packet is not fragmented, remaining 13 bits indicates fragment offset. 25th byte “01000000” or “0x40” or “Decimal 64” indicates TTL value. 26th byte “00000110” or “0x06” indicates that this is a ‘tcp’ packet. 27th and 28th packets indicates Header Checksum. 29th to 32nd bytes indicates source ip address (127.0.0.1) and 33rd to 36th byte indicates destination ip address (127.0.0.1). This ends the ‘ip header’, we are now moving to ‘tcp’ header (Transport layer in OSI).

TCP Header

To understand details from 37th byte to 76th byte, we need to refer ‘TCP Header‘. 37th and 38th bytes “11100101 01100011” or “decimal 58723” indicates the source port number. 39th and 40th bytes “00010101 10110011” or “decimal 5555” indicates destination port number (means incoming packet is trying to connect port 5555). 41st to 44th byte indicates sequence number and 45th to 48th byte indicates sequence acknowledgement number. Higher order 4 bits in 49th and 50th byte “1010” or “Decimal 10” indicates Data offset, means there are 10*4=40 bytes in TCP header. Next 3 higher order bits are reserved in 49th byte 50th byte. Next 3 bits indicates ECN. Next 6 bits “000010” or “0x02” indicates that ‘SYN’ flag was set in Control bits. 51st and 52 bytes indicates window size, means the sender is willing to accept “10000000 00011000” or “decimal 32792” bytes in the response packet. 53rd and 54th bytes indicates checksum. 55th and 56th bytes indicates Urgent pointer, usually 0.

Options

Inside TCP header, bytes 57 to 76 contains value based on 50th byte(Data Offset). In this particular packet, 50th byte has (0xa), which means, TCP header in this packet contains totally 40bytes. Mandatory TCP fields (from 37th byte to 56th byte) are already discussed, but we have 20 more bytes to decode, these bytes are represented as ‘Options’ in TCP header. They may occur or they may not occur in a TCP packet. Mostly they occur in SYN packet.

Here, 57th byte (0x02) represents option-kind, 58th byte represents option-length (0x04). Both 57th and 58th bytes represents that 59th and 60th bytes “01000000 00001100” or “0x400c” or “Decimal 16396” indicates “Maximum Segment Size“. 61st byte (0x04) represents option-kind, 62nd byte (0x02) represents option-length, both bytes represents “SACK permitted“. 63rd byte (0x08) represents option-kind, 64th byte (0x0a) represents option-length, both bytes indicates that from 65th byte to 68th byte contains ‘TSVal‘ and from 69th byte to 72nd byte contains ‘TSecr‘. 73rd byte “0x01” indicates option-kind as ‘No-Operation‘, 74th byte (0x03) indicates option-kind, 75 byte indicates (0x03) option-length, both bytes indicates that 76th byte (0x05) contains ‘WSOpt (Window Scale Option)‘, which means, the host which sent this packet can accept upto “32792 * (2^5)” or (windowsize[byte51&52] * (2^wsopt[byte76])) before sending ACK.

pcapstreamer with awk

To display only ICMP packets, we can use the following commandline

$ sudo ./pcapstreamer 2>/dev/null | awk '{if($26 ~ "00000001"){print $0;}}'

To display only SYN packets, we can use the following commandline

$ sudo ./pcapstreamer 2>/dev/null | awk '{ctrlbytes=$49$50; if(ctrlbytes ~ "^.......000010...$"){print $0;}}'

I hope this utility may be useful for newbies like me to learn networking. Have a great new year.

Advertisements

Systemd for Simple Backup

Hi,

Nice to get back, Recently switched to F15. Wow!! my userland changed heavily. Previously, its very simple things like,

SysVinit for booting
Udev for devices
pppconfig for Network
Xfce for GUI
ALSA for sound
ffmpeg & mplayer for video

Now,

Systemd for booting, daemons
Udev, DBus, UDisks for devices
ModemManager+NetworkManager for Network
Gnome3 for GUI
PulseAudio+ALSA for sound
GStreamer+totem for video

These Technologies are very interesting to learn. I’m not going to explain about each. But, I went through systemd and came up with a solution for my backup problem.

Backup Problem:

While in its last phase, my previous laptop teached me the importance of external backup disks. So, I bought an external 512GB Segate, and backed-up /home/${HOME} tree. It helped me to quickly get back my files to F15. However, one problem is, maintaining my backup. I thought If I plug my external drive, someone should automatically copy all the new files resides in my current /home/${HOME} in F15 to that drive. There are lot of ways to do it. I can think of two main ways,

* Write an udev rule to call a script which will mount that external backup partition and rsync /home/${HOME} in F15 to that external partition. Pros: Fairly straight forward, udevrulefile+rsyncscript will do it. Cons: No control, this will copy every time udev detects that external drive. If there is lot of files to copy, then this will make a mess.

* Use systemd to call a script whenever that external backup partition gets mounted. Pros: You have full control, create a backup.service for systemd, create backup.bash to rsync /home/${HOME} to backup disk. Enable that service in systemd to automatically do rsync, or just load that service to systemd and only start that service If you want to backup your files. Cons: Need to pass one more layer to run the actual rsync script.

So, I took-up systemd

There are lot of ways you can trigger your service in systemd. Also, you can depend on another systemd unit to trigger your service. In this case, I depend on a mount unit to trigger my backup.service. This service, in-turn, will trigger backup.sh script.

To define a systemd unit, you need to know what type of unit you want to create. Currently there are 10 types of units systemd can understand [read systemd.unit(5)]. For backup job, I used two type units, one is systemd.mount(5) amd systemd.service(5). If you go through systemd.mount(5) you will understand that systemd will automatically load this units whenever systemd saw a block device. So, systemd will automatically provide ‘media-ExternalBackupPartition.mount’ whenever I insert my external hard disk . I only need to define the next unit, backup.service for systemd.

Note: There is a story behind the name ‘media-ExternalBackupPartition’. I’ll tell you at the end of this post. Lets just continue with systemd for now.

To define a unit for systemd, you need to create a file as /etc/systemd/system/name.unit; (for my backup problem, unit file is /etc/systemd/system/backup.service). Here ‘name’, may be anything relevent to your job and ‘unit’ must be one of ‘service’, ‘socket’, ‘device’, ‘mount’, ‘automount’, ‘swap’, ‘target’, ‘path’, ‘timer’ and ‘snapshot’. Read systemd.unit(5) man pages for more precise information.

Unit Class:
Unit definition files contains information in .ini format. One of the class ‘[Unit]’ must exist in every unit file. Here is the [Unit] class for backup.service

[Unit]
Description='sync my files with external backup drive'
Requires=media-ExternalBackupPartition.mount
After=media-ExternalBackupPartition.mount

Here ‘Description’ is a general description for your service. We need to put the required unit which will trigger this new unit in ‘Requires’ field. Systemd will run this unit once all the units in ‘Requires’ fields satisfied. We can call these ‘Requires’ units as parent units, and your unit as child. However, systemd will not wait for parent units to complete to run child unit. We need to explicitly ask systemd to wait for parent units to complete, For this purpose we have ‘After’ field. If you define which parent unit needs to be completed before your child unit could run, you need to mention it in ‘After’ field.

For my backup.service child unit, ‘media-ExternalBackupPartition.mount’ unit must be satisfied in systemd. That means, my external HD partition must be mounted inside ‘/media/ExternalBackupPartition’ path. Also, Using ‘After=’ field, I instructed systemd, not to start this child unit before ‘mount-ExternalBackupPartition.mount’ finishes.

Service Class:

Now we need to define what to do once the requirements satisfies. For that purpose, we need to define ‘[Service]’ class, Here is the service class for backup.service,

[Service]
Type=simple
ExecStart=/home/mohan/Development/scripts/backup.sh

This ‘[Service]’ class is specific to ‘systemd.service’ units. Here, ‘ExecStart=/home/mohan/Development/scripts/backup.sh’ asks systemd to run ‘backup.sh’ whenever ‘backup.service’ satisfies. You can do lot of customization to setup the execution environment before start running any commands, such as log redirection, demonizing etc., there are lot of fields to use in ‘[Service]’ class, but I simply used ‘Type=simple’ to tell systemd, that no need to do any change in execution environment. The script ‘backup.sh’ will take care of all the redirection within itself.

Install Class:
In SysV init system, we use ‘chkconfig (redhat/fedora)’ or ‘update-rc.d (debian)’ to enable or disable a service. In systemd, we use this ‘[Install]’ class to enable or disable our ‘backup.service’ unit so that it will work even after a restart. Here is install class for backup.service,

[Install]
WantedBy=media-ExternalBackupPartition.mount

In systemd, enabling a service means, adding a symlink to ‘/etc/systemd/system/name.service.wants/’ directory. disabling a service means, removing that symlink. ‘systemctl’ command can do this add/remove symlink automatically when we call it with ‘systemctl enable backup.service’ or ‘systemctl disable backup.service’, but we need to say the parent unit name, thats why we have this ‘[Install]’ class. Simply, we need to mention that parent unit in ‘WantedBy=’ field.

Execution:

Once the unit files are ready, we need to enable them into systemd. For backup.service, I executed following commands to setup the service

$ sudo cp ~/backup.service /etc/systemd/system/backup.service
$ sudo systemctl daemon-reload

I just copied backup.service unit file to systemd’s location and asked systemd to reload unit definitions. Now we can check if things loaded properly or not using following command,

$ sudo systemctl status backup.service
backup.service - 'sync my files with external backup drive'
          Loaded: loaded (/etc/systemd/system/backup.service)
          Active: inactive (dead)
          CGroup: name=systemd:/system/backup.service
$

systemd will say ‘Loaded : error’ if it can’t understand any defnintion in backup.service or if it can’t satisfy the definitions. Otherwise we can start this service using following command, we need to make sure the final rsync script ‘backup.sh’ exists in the location pointed by ‘ExecStart=’ field.

$ sudo systemctl start backup.service

This will start syncing new files to External backup HD drive, only when It is plugged-in and mounted. Otherwise, the service will fail. you can check the status again using ‘systemctl status’. Once you checked that the service is working as intended, we can enable this service (I mean, creating symlinks) using following command,

$ sudo systemctl enable backup.service

We can verify the symlink as below to make sure parent-child linking is done correctly.

$ ls -l /etc/systemd/system/media-ExternalBackupPartition.mount.wants/baskup.service
lrwxrwxrwx 1 root root 34 Nov  9 02:08 /etc/systemd/system/media-ExternalBackupPartition.mount.wants/backup.service -> /etc/systemd/system/backup.service

If we don’t want to copy automatically, we can disable it using below command,

$ sudo systemctl disable backup.service

Even If the service is disabled, you can start/stop the service. Systemd will recognize the backup.service, check it’s dependencies and execute ‘backup.sh’ correctly.

media-ExternalBackupPartition.mount:

As I already said, systemd will automatically create units using udev, so when my external HD plugs-in, udev will tell to udisks that a new partition is available, then udisks will mount that partition inside ‘/media/uuid’ location, then systemd will create a unit as ‘media-uuid.mount'(systemd uses ‘-‘ instead of ‘/’ for path seperation). But specifying ‘Requires=media-uuid.mount’ inside backup.service file is not working. Thus, I used a simple udev rule to rename udisk’s mount path, here is the rule file,

$ cat 99-rename-udisk-mountpoint.rules
ENV{ID_FS_UUID}=="251c683d-bce0-489c-aab5-f684a9a1f3b2",ENV{ID_FS_UUID}="ExternalBackupPartition"
$ sudo cp ~/99-rename-udisk-mountpoint.rules /etc/udev/rules.d

This above two commands, will modify udisk’s mount path to ‘/media/ExternalBackupPartition’ instead of ‘/media/251c683d-bce0-489c-aab5-f684a9a1f3b2′, thus systemd will automatically create ‘media-ExternalBackupPartition.mount’ instead of ‘media-251c683d-bce0-489c-aab5-f684a9a1f3b2.mount’

Finally, Thanks for reaching this line. I hope this long boring article will help you to understand something about systemd.

Creating Fedora DVD Repo

Hi,

Its almost 1 year since my last post, what happens to me? It a 1 year story to say. But in short, my laptop became older and eventually went down. Graphics card gone. It took a while for me to buy another new one. Here I am.

In between lot thing happened, Something I just wanted to post, but now I can’t remember them. As a fresh start, This is what happened when I got my new ‘Samsung Notebook’. Ordered through FlipKart and got it by last week.

I know that I can’t get Internet because I cancelled my netconnect+ subscription way back when my previous lappy started giving troubles. Thus, My beloved debian is out of reach. So, I decided to get Fedora or Mint DVD from someone and Install it. Thanks to Srini, Amachu and his Gang, for providing Fedora DVD to me. Installation went smoothly and for the first time, I’m in Gnome3,

Fedora 15 Desktop
Fedora 15 Desktop

Installation went smoothly and to my surprise, no more tinkering on mobile modem. Thanks to those guys who created profile for Aircel. At first, bluetooth daemon didn’t start, googled and fixed it.

While playing with Gnome3, I thought of using installation DVD as my repo, because I have limited network with very little speed, It would be better to use existing DVD as my repo. So, that starts my hunger and this is how I ended.

1. Inserted my DVD and monitored how my DVD is mounting. Seems ‘org.freedesktop.UDisks’ takes care of mounting and it always mount my DVD inside ‘/media’ folder taking mount directory name from ‘ENV{ID_FS_LABEL_ENC}” Udev environment variable. I came to know that yum is facing some trouble If I have a ‘fedora-dvd.repo’ file like below under ‘/etc/yum/yum.repos.d’ directory,

[fedora-dvd]
name=Fedora DVD $releasever - $basearch
baseurl=file:///media/Fedora 15 i386 DVD
enabled=1
gpgcheck=1
gpgkey=file:///media/Fedora 15 i386 DVD/RPM-GPG-KEY-fedora-$basearch

So, it seems yum cannot handle white-space in URI. Then, I decided to write a Udev rule to rename ‘Fedora 15 i386 DVD’ to ‘Fedora15i386DVD’, So that ‘org.freedesktop.UDisks’ will mount it as ‘file:///media/Fedora15i386DVD’. Here is the rule file

$ cat /etc/udev/rules.d/99-rename-fedora-dvd.rules
ENV{ID_FS_LABEL_ENC}=="Fedora\x2015\x20i386\x20DVD", ENV{ID_FS_LABEL_ENC}="Fedora15i386DVD"
$

Then, I recreated fedora-dvd.repo as below

$ cat /etc/yum.repos.d/fedora-dvd.repo
[fedora-dvd]
name=Fedora DVD $releasever - $basearch
baseurl=file:///media/Fedora15i386DVD
enabled=1
gpgcheck=1
gpgkey=file:///media/Fedora15i386DVD/RPM-GPG-KEY-fedora-$basearch
$

One of the main reason to add this ‘Fedora15i386DVD’ repo is to install development tools without internet. In a way it is more useful than a plain LiveCD.

Don’t forget to insert Fedora DVD If you follow this setup and want to install something from Fedora DVD. A little google tells me that there are lot of ways to achieve this DVD-to-Repo thing in Fedora, I’m relatively new to Fedora, I will learn as I move on.