Fetchmail

As you already know from by previous postfix article, Any mail going through drunkenmonk.org domain will have my gmail address as sender. Thats good. What about receiving mail from gmail?

In your mind, you are saying “Its damn easy right? configure your evolution/thunderbird to poing to gmail stupid?” right? yeah, thats right. I can configure any mail client to point to gmail and get mails. But, I want my postfix to play a roll.

Postfix is basically a smtp server, it can also act like smtp client. But, it cant work as a imap/pop3 client.

So, Here is the task, I need to get mails from gmail through pop3 and put it into my postfix so that all other clients connected with my postfix can get that mail. Here comes fetchmail.

Fetchmail is a beautiful program which will act like imap/pop3 client (just like evolution/thunderbird) and instead of saving it somewhere, fetchmail will put that mail into your local machine’s mail server. So, all the mail clients connected with your local mail server will get your email from gmail.

Thats enough of theory, lets get down and install fetchmail, first install the package

$ sudo equo install fetchmail
$

Now time to configure fetchmail, it uses /etc/fetchmailrc as its only configuration file. But, we need to be careful with it. Because, we are saving the username/password a plain-text, so the permission for /etc/fetchmail should be (0600) for root.

If you start configuring fetchmail by reading ‘man fetchmail’, you will switchoff easily. That man page is such a damn brief and will confuse you easily. But, fetchmail config is really a easy one. It will take just three lines to configure it. Here is those three lines,

set daemon 120
set syslog
poll pop.gmail.com proto pop3 user "foo" pass "bar" to "stupid" ssl sslproto "TLS1"

Here, ‘set daemon 120’ will make fetchmail to connect to gmail server every 2 minutes.

‘set syslog’ will make fetchmail to put logs into syslog.

‘poll pop.gmail.com’ instructs fetchmail to connect to pop.gmail.com server.

‘proto pop3’ instructs fetchmail to use pop3 protocol.

‘user “foo” pass “bar” to “stupid”‘ instructs fetchmail to use ‘foo’ as username and ‘bar’ as password to open gmail account in gmail server, fetch those mail from gmail server and put it into local machine’s postfix as a mail for ‘stupid’ user.

‘ssl’ instructs fetchmail to use secure connection. ‘sslproto “TLS1″‘ tells fetchmail to use STARTTLS for sercure connection.

Thats it for configuration stuff, now its time to restart fetchmail

$ sudo eselect rc restart fetchmail
$

Now, you can see your mails are fetched from gmail and available to you through ‘mailx’ command.

Dovecot (with mbox)

There are different ways to save a mail in *nix systems. The old baddy is mbox which saves all your email in one single file. The new one is Maildir.

Even though mbox have difficulties, traditional commandline mail clients like ‘mailx’ works with ‘mbox’ style mail files flawlessly. Also postfix which I setup previously by default creates single mbox mail file inside /var/spool/mail/ directory for each user. So, I need a imap+pop3 server which can operate with mbox like mail files. Dovecot is the choice.

First we need to install dovecot. I’m using sabayon, the instruction is specific to sabayon/gentoo,

$ sudo equo install net-mail/dovecot

Its time to setup dovecot. Before we do, we need to create self signed certificates to use it for imaps and pop3s. Lets first create those certificates.

$ cd /etc/ssl/dovecot
$ sudo mkdir oldcerts
$ sudo mv * oldcerts
$ sudo openssl genrsa -out server.key.password -des3 1024
Generating RSA private key, 1024 bit long modulus
..........................................++++++
..................................................++++++
e is 65537 (0x10001)
Enter pass phrase for server.key.password:
Verifying - Enter pass phrase for server.key.password:
$

We have to give ‘pass phrase’ here, otherwise openssl command will not create key file. But, having pass phrase for key file is not good, because we need to provide this pass phrase everytime dovecot access this file. Here is the step to remove the pass phrase from rsa key file

$ sudo openssl rsa -in server.key.password -out server.key
Enter pass phrase for server.key.password:
writing RSA key
$

Now generate certificate request with the rsa key file

$ sudo openssl req -out server.csr -new -key server.key
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:TamilNadu
Locality Name (eg, city) []:Chennai
Organization Name (eg, company) [Internet Widgits Pty Ltd]:DrunkenMonk Private Limited
Organizational Unit Name (eg, section) []:Pattasarayam Generating Unit
Common Name (e.g. server FQDN or YOUR name) []:drunkenmonk.org
Email Address []:webmaster@drunkenmonk.org

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
$

We should not give password when creating certificate request. Ok, know time to create self signed certificate.

$ sudo openssl x509 -req -in server.csr -out server.crt -signkey server.key -days 365
Signature ok
subject=/C=IN/ST=TamilNadu/L=Chennai/O=DrunkenMonk Private Limited/OU=Pattasarayam Generating Unit/CN=drunkenmonk.org/emailAddress=webmaster@drunkenmonk.org
Getting Private key
$

Now we have two files /etc/ssl/dovecot/server.key and /etc/ssl/dovecot/server.crt to use it for SSL. Lets configure dovecot now, we need to modify /etc/dovecot/dovecot.conf like this,

protocols = imap pop3
listen = *, ::

Configuration not ends with /etc/dovecot/dovecot.conf, it has different conf files for different purpose inside /etc/dovecot/conf.d/, lets modify one by one, Here is /etc/dovecot/conf.d/10-auth.conf

auth_mechanisms = plain login
!include auth-system.conf.ext
#!include auth-sql.conf.ext
#!include auth-ldap.conf.ext
#!include auth-passwdfile.conf.ext
#!include auth-checkpassword.conf.ext
#!include auth-vpopmail.conf.ext
#!include auth-static.conf.ext

If you want to permit only current machine’s users and don’t want to use ldap or other machinisms, then make sure you comment all includes except auth-system.conf.txt.

Here is modifications inside /etc/dovecot/conf.d/10-logging.conf

log_path = syslog
syslog_facility = mail

Here is modifications inside /etc/dovecot/conf.d/10-mail.conf

mail_location = mbox:~/.mail:INBOX=/var/mail/%u

Now /etc/dovecot/conf.d/10-ssl.conf

ssl = yes
ssl_cert = /etc/ssl/dovecot/server.crt
ssl_key = /etc/ssl/dovecot/server.key

Thats all from config stuff, lets restart dovecot.

$ sudo eselect rc restart dovecot

Now, we need to test if it is working, We need to connect with dovecot through imap port with TLS encryption

$ openssl s_client -connect localhost:143 -starttls imap

Above command will show lot of SSL stuffs, and then finally dovecot will say ‘. OK’, we need to start communication with dovecot from there,

. OK Pre-login capabilities listed, post-login capabilities have more.
a login mokka somepassword
* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE
a OK Logged in
b select inbox                   
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1343790396] UIDs valid
* OK [UIDNEXT 320] Predicted next UID
* OK [NOMODSEQ] No permanent modsequences
b OK [READ-WRITE] Select completed.
c list "" *
* LIST (\HasNoChildren) "/" "INBOX"
c OK List completed.

The above session with dovecot shows that things are ok. You can now configure your Thunderbird/Evolution to use your machine for local emails. Have a nice day!!

Postfix (with smtp.gmail.com as relayhost)

Most of us forget that postfix not only acts like a mail-server, but it can act like a smtp-client (like thunderbird, evolution).

This post is all about configuring postfix to work as mail-server for my local domain (drunkenmonk.org) at the same time forward any outside domain mails to smtp.google.com with my gmail account credentials. If you are curious about drunkenmonk.org, read my previous articles about dnsmasq and virtual home network. These two articles will explain how I setup my home network.

First things first, lets install postfix (Well I’m using sabayon)

$ sudo equo install mail-mta/postfix

BASIC MAIL SERVER

Postfix configuration is pretty simple and straight forward. You have to configure following parameters inside /etc/postfix/main.cf

mydomain = drunkenmonk.org
myorigin = $mydomain
inet_interfaces = all
mydestination = localhost, localhost.$mydomain, $myhostname, $mydomain, mail.$mydomain
mynetworks_style = subnet

‘mydomain’ tells postfix that it should serve as mail server for ‘drunkenmonk.org’ domain.

‘myorigin’ tells that postfix should append ‘drunkenmonk.org’ to any incoming mail’s address if the address don’t have domain part. Which means, when you do ‘mail someuser’ from commandline, postfix will append ‘drunkenmonk.org’ so that To address will become ‘someuser@drunkenmonk.org’.

‘inet_interfaces’ tells postfix to listen on all network interfaces.

‘mydestination’ tells postfix that it should be the final destination for ‘localhost’, ‘localhost.drunkenmonk.org’, ‘mokka.drunkenmonk.org’, ‘drunkenmonk.org’ and ‘mail.drunkenmonk.org’

‘mynetworks_style’ tells postfix that my local network is a ‘subnet’ type.

Thats all needed for postfix to get start. We can restart postfix with this configurations to send/receive mails between local users. However we want a little bit more from postfix. Lets configure it to use STARTTLS for its communication with its clients

BASIC MAIL SERVER WITH STARTTLS

If you wonder what is STARTTLS, its a simple method to establish communication in a secure way. Your server application (like Postfix listening on port 25) says “Hey client I have TLS support, If you want, we can communicate each other in TLS secure way within port 25 itself, If you don’t want, we can continue with plain-text, what you say?”, then your client (like Evolution or Thunderbird) will tell “OOh!! you offer TLS, then lets do TLS!!”. So the packets going between server and client will be encrypted using TLS.

Before STARTTLS, people used different port for secure smtp which is smtps(port 465). But nowadays it is changing, the regular smtp port itself been used for both plain-text and SSL/TLS communication.

We have to configure Postfix to talk in SSL/TLS way, for that we need Self Signed SSL certificate. Lets create one,

$ cd /etc/ssl/postfix
$ sudo mkdir oldcerts
$ sudo mv * oldcerts
$ sudo openssl genrsa -out server.key.password -des3 1024
Generating RSA private key, 1024 bit long modulus
.............................++++++
..........................++++++
e is 65537 (0x10001)
Enter pass phrase for server.key.password:
Verifying - Enter pass phrase for server.key.password:
$

Here you have to give a ‘pass phrase’ to generate RSA private key, but having a key with ‘pass phrase’ is asking for trouble. You have to manually type this password everytime postfix access this key. The following command will remove ‘pass phrase’ from a RSA key

$ sudo openssl rsa -in server.key.password -out server.key
Enter pass phrase for server.key.password:
writing RSA key
$ 

Now you have a RSA private key file server.key without ‘pass phrase’. Its time to create certificate request,

$ sudo openssl req -out server.csr -new -key server.key
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:TamilNadu
Locality Name (eg, city) []:Chennai
Organization Name (eg, company) [Internet Widgits Pty Ltd]:DrunkenMonk Pvt Ltd
Organizational Unit Name (eg, section) []: Pattasarayam Generating Unit
Common Name (e.g. server FQDN or YOUR name) []:MokkaPandi
Email Address []:webmaster@drunkenmonk.org

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Don’t give any password when creating certificate request. Finally create a self signed ssl certificate using following command,

$ sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=IN/ST=TamilNadu/L=Chennai/O=DrunkenMonk Pvt Ltd/OU= Pattasarayam Generating Unit/CN=MokkaPandi/emailAddress=webmaster@drunkenmonk.org
Getting Private key
$

Ok, thats it. You will have two files, one is /etc/ssl/postfix/server.crt and another is /etc/ssl/postfix/server.key, Now configure postfix to use these two for SSL/TLS communication, Add the following lines at the end of /etc/postfix/main.cf

# TLS Support
smtpd_tls_cert_file = /etc/ssl/postfix/server.crt
smtpd_tls_key_file = /etc/ssl/postfix/server.key
smtpd_tls_security_level = may

Finally, start postfix

$ sudo eselect rc restart postfix

Check if postfix can operate with STARTTLS,

$ sudo openssl s_client -connect localhost:25 -starttls smtp

Above command will show the server certificate and all SSL stuffs. After this, whatever you send will be encrypted, now check if it can send mail,

250 DSN
EHLO drunkenmonk.org
250-mokka.drunkenmonk.org
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: root@drunkenmonk.org
250 2.1.0 Ok
rcpt to: webmaster@drunkenmonk.org
250 2.1.5 Ok
data
354 End data with .
Subject: test mail
test mail
.
250 2.0.0 Ok: queued as E0E352C0091
quit
221 2.0.0 Bye
closed
$

Now see if root got a test mail

$ sudo mail
>N  8 root@drunkenmonk  Tue Aug  7 16:50   14/533   test mail
&p
From root@drunkenmonk.org  Tue Aug  7 16:50:23 2012
X-Original-To: webmaster@drunkenmonk.org
Subject: test mail
Date: Tue,  7 Aug 2012 16:49:48 +0530 (IST)
From: root@drunkenmonk.org

test mail

&q
$

We are not done yet. There is one more part. Lets make postfix to forward mails to gmail.

MAIL FORWARDER

Till now, postfix can send/receive mails within drunkenmonk.org domain. But if someone from your subnet want to send a mail to a person who have mail address outside drunkenmonk.org domain (let say stuid@yahoo.com), then your current postfix will try to contact mail server of yahoo.com. But, yahoo’s mail server will reject your mail because drunkenmonk.org is not a vaild registered domain.

But, If you have a gmail account, you can forward the mail first to smtp.gmail.com with your account credentials so that gmail will send your mail to stupid@yahoo.com with your gmail account credentials. The person who owns stupid@yahoo.com will receive a mail with from address as yours irrespective of whoever send within your local network.

To make postfix to know your credentials, we need to create a credential db with gmail account credentials, For that we need to add following line at the end of /etc/postfix/saslpass file

# run following command to generate hash password db
# that can be used inside postfix's main.cf
#
# command:
# postmap hash:saslpass
#
#
#file content format:
#remotehost username:password
smtp.gmail.com foouser:barpassword

Now run the following command to create saslpass.db

$ cd /etc/postfix
$ sudo postmap hash:saslpass

Now you can see saslpass.db created as a Berkeley DB file. We can now use this file inside main.cf to inform postfix about your gmail account credential. Before doing that, please remove /etc/postfix/saslpass file or change its access to 0600 for root. Because your credentials are in plaintext. We need to add following lines at the end of main.cf

# gmail
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpass
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt
smtp_tls_mandatory_ciphers = high
smtp_tls_verify_cert_match = hostname, nexthop, dot-nexthop
relayhost = smtp.gmail.com:587

Here ‘hash:/etc/postfix/saslpass’ actually makes postfix to access /etc/postfix/saslpass.db file. You can remove /etc/postfix/saslpass which have your gmail password as plaintext. It will not affect postfix and postfix doesn’t need it.

Thats it, we are done. Once you restart postfix, It becomes capable of sending mail to outside world but with your gmail account’s credentials. Thanks for reading all the way down to this line. Have a nice day!!

Virtual Home Network (with Dnsmasq , Bridge, TUN/TAP, Qemu)

As I said in my previous article, this post will explain how we can use bridge interface to configure networks inside virtual hosts and much more about virtualization.

First we need to make sure our machine is capable of kvm virtualization, see if you get any output for below command

$ grep -E 'vmx|svm' /proc/cpuinfo

Most modern processers supports Hardware assisted Virtualization. If you don’t get any output for above command, It means you don’t have a processor capable of providing Hardware Assisted Virtualization. Qemu-kvm will not work, but you can use Qemu without kvm or some other virtualization applications like virtualbox etc. Also skip ‘LOADING KERNEL MODULES’ section of this article if you don’t have kvm support.

LOADING KERNEL MODULES

If your processor supports kvm, then we have to load following kernel modules and make sure they load automatically whenever we restart our system. In gentoo/sabayon, we need to modify ‘modules=’ line in /etc/conf.d/modules file as

modules="kvm-intel tun"

Here ‘kvm-intel’ is the kernel module for virtualization and ‘tun’ is the kernel module for ‘TUN/TAP’ devices. For, AMD machines, we need to load ‘kvm-amd’ instead of ‘kvm-intel’. Now we need to load these modules for the current run, Here is the commands which will load ‘kvm-intel’ and ‘tun’ drivers into kernel

$ sudo modprobe kvm-intel tun
$

INSTALLING PACKAGES

Next is to install required packages, Here is sabayon command to install required packages

$ sudo equo install app-emulation/qemu-kvm net-misc/bridge-utilities net-dns/dnsmasq sys-apps/usermode-utilities
$

‘qemu-kvm’ is for virtualization, ‘bridge-utilities’ is to get ‘brctl’ command, ‘dnsmasq’ is to handle DNS and DHCP requests from guests, ‘usermode-utilities’ is to get ‘tunctl’ command.

CREATING INTERFACES

Its time to create interfaces, first we need to create the bridge

$ sudo brctl addbr br0

Assign IP address to the bridge,

$ sudo ifconfig br0 192.168.2.1 netmask 255.255.255.0 up

Next, create a tap0 interface, following command will create ‘tap0’ pseudo slave interface from /dev/net/tun master interface

$ sudo tunctl -t tap0

Now, we need to hook the tap0 interface with the bridge br0

$ sudo brctl addif br0 tap0

Bring up the tap0

$ sudo ifconfig tap0 up

FIREWALL RULES AND MASQUERADING

If you read my previous article, I have dnsmasq configured to serve DHCP requests coming from br0. Hooking tap0 into br0 will make the guest send DHCP requests via tap0->br0 and my host machine’s dhsmasq process will serve ip address to the guest. We have to make sure iptables dont block BROADCAST packets as well as open the port 53 and 64 so that dnsmasq will get the DHCP and BOOTP packets. Make sure your iptables contains following lines,

$ sudo iptables -t filter -L | grep -E 'BROADCAST|domain|bootp'
ACCEPT     all  --  anywhere             anywhere             ADDRTYPE match src-type BROADCAST
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domainflags: FIN,SYN,RST,ACK/SYN
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootpsflags: FIN,SYN,RST,ACK/SYN
$

Now we need to make sure MASQUERADING is enabled and let your host system forward the packets. Also make these settings permanent so that reboot dont break the settings,

$ sudo iptables -t nat -A POSTROUTING -j MASQUERADE
$ sudo sysctl net.ipv4.ip_forward=1
$ sudo /etc/init.d/iptables save
$ sudo /etc/init.d/iptables reload
$ sudo sed -i 's/^.*net.ipv4.ip_forward.*$/net.ipv4.ip_forward = 1/g' /etc/sysctl.conf

Check these settings are ok,

$ sudo iptables -t nat -L
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere
$ sudo sysctl -a | grep ipv4.ip_forward
net.ipv4.ip_forward = 1
$

STARTING QEMU GUEST

Finally, its time to start virtual guest, first we need to create disk image, Here is the command to create a 5GB raw disk image for qemu-kvm

$ sudo qemu-img create -f raw Debian.img 5G
$

Now download debian netinstall image for amd64 architecture

$ sudo wget http://ftp.nl.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/netboot/gtk/mini.iso
$

Atlast, start the guest with with tap0 networking

$ qemu-kvm -cpu kvm64 -drive file=Debian.img,if=virtio -cdrom mini.iso -boot order=dc -m 512 -soundhw sb16 -name "Debian" -net nic,model=virtio -net tap,ifname="tap0",script=no,downscript=no

Thats it, your qemu guest will automatically get network settings from host machine. You can also verify that dnsmasq served a DHCP request via syslog

$ sudo grep 'dnsmasq-dhcp' /var/log/messages
dnsmasq-dhcp[9810]: DHCPDISCOVER(br0) 192.168.2.89 52:54:00:12:34:56
dnsmasq-dhcp[9810]: DHCPOFFER(br0) 192.168.2.89 52:54:00:12:34:56
dnsmasq-dhcp[9810]: DHCPREQUEST(br0) 192.168.2.89 52:54:00:12:34:56
dnsmasq-dhcp[9810]: DHCPACK(br0) 192.168.2.89 52:54:00:12:34:56 virt0
$

Well, if you still reading this article and not get bored, you must be a *nix admin. Have a great day!!

Dnsmasq for Home User

Every BSNL broadband subscribers know that BSNL’s default nameservers are one of the worst in response times. Even if you have connectivity exceeding 1gbps, If your nameservers are not good, then your internet experience will not be good.

So, I decided to use google’s nameservers. I could have configure NetworkManager not to get nameservers through DHCP instead ask it to use google’s nameservers, but I thought of giving ‘dnamasq’ a try.

For those who don’t know what is mean by dnsmasq, it is a dns forwarder which can handle DNS, DHCP and BOOTP requests. It means, it can act like bind, dhcpd and tftpd.

Configuring dnsmasq is pretty simple and straight forward. We have to modify two files /etc/hosts and /etc/dnsmasq.conf. Here is my /etc/hosts file

127.0.0.1     mokka.drunkenmonk.org mokka localhost
::1           mokka.drunkenmonk.org mokka localhost
192.168.2.1   bridge.mokka.drunkenmonk.org

The first line says ‘127.0.0.1 is the ip address for mokka.drunkenmonk.org and also for ‘mokka’ and ‘localhost’ hostnames. I other words, ‘mokka’, ‘localhost’ and ‘mokka.drunkenmonk.org’ will resolve to 127.0.0.1 ip address.

‘mokka.drunkenmonk.org’ is a cononical name which contains two parts, ‘mokka’ as hostname and ‘drunkenmonk.org’ as domainname. So, by putting ‘drunkenmonk.org’, I’m also setting domainname for my system.

I really don’t know the proper way to configure domainname in my sabayon (in other words gentoo). From googling, I came to know that putting ‘domain drunkenmonk.org’ into /etc/resolv.conf is the right way to set domainname in a linux system. But, /etc/resolv.conf is such dynamic nowadays, we can’t be sure who will modify it later and I never see a router providing domainname in DHCP response (Not in BSNL’s ADSL routers for sure). So I endup putting my domainname in /etc/hosts file.

The second line is same like the first one, but for ipv6.

The third line says ‘192.168.2.1’ is the ip address for ‘bridge.mokka.drunkenmonk.org’. I’ll come to this later. Now, take a look at my /etc/dnsmasq.conf

domain-needed
bogus-priv
no-resolv
server=8.8.8.8
server=8.8.4.4
local=/drunkenmonk.org/
domain=drunkenmonk.org
dhcp-range=interface:br0,192.168.2.2,192.168.2.254,255.255.255.0,1d
mx-host=drunkenmonk.org,mail.drunkenmonk.org,30
cname=drunkenmonk.org,bridge.mokka.drunkenmonk.org
cname=www.drunkenmonk.org,bridge.mokka.drunkenmonk.org
cname=mail.drunkenmonk.org,bridge.mokka.drunkenmonk.org

‘domain-needed’ instructs dnsmasq to never forward DNS queries which don’t have domain part. It means, If you query for ‘nslookup google’ dnsmasq will never forward it to upstream server (In my case, to google’s nameservers).

‘bogus-priv’ tells dnsmasq never forward reverse-lookup queries which have local subnet’s ip rage to upstream. Which means ‘nslookup 10.0.0.1’ will not be forwarded to upstream instead dnsmasq will try to resolv itself. If it doesn’t find hostname for 10.0.0.1 in /etc/hosts or its dhcp leases, then it will send back ‘no such domain’ response.

‘no-resolv’ says dnsmasq will not read /etc/resolv.conf to get upsteram nameservers. Normally dnsmasq will read /etc/resolv.conf file to get upstream nameservers.

‘server=8.8.8.8’ and ‘server=8.8.4.4’ instucts dnsmasq to use google’s 8.8.8.8 and 8.8.4.4 as primary and secondary DNS nameservers (or in other words upstream nameservers).

‘local=/drunkenmonk.org/’ says that DNS queries with hostnames like ‘mokka.drunkenmonk.org’ should not be forwarded to upstream intead dnsmasq should resolve it from /etc/hosts or from its dhcp leases.

‘domain=drunkenmonk.org’ tells dnsmasq to send ‘domain=drunkenmonk.org’ in DHCP response so that machines configured through DHCP will come under ‘drunkenmonk.org’ domain.

‘dhcp-range=interface:br0,192.168.2.2,192.168.2.254,255.255.255.0,1d’ instructs dhsmasq to allocate ip address between 192.168.2.2 and 192.168.2.254 with netmask 255.255.255.0 for 1 day only to the DHCP requests coming from br0 interface. Which means, machines connected through br0 interface and asking for DHCP response will get ip address between 192.168.2.2 and 192.168.2.254 for 1 day. dnsmasq will not send response to requests coming from interfaces other than br0.

‘mx-host=drunkenmonk.org,mail.drunkenmonk.org,30’ tells dnsmasq to send ‘mail.drunkenmonk.org’ as MX response for ‘drunkenmonk.org’ domain. Which means, when you do ‘nslookup -q=MX drunkenmonk.org’ it will give ‘mail.drunkenmonk.org’ as response.

‘cname’ lines are like aliases. means when you do ‘nslookup http://www.drunkenmonk.org’ it will resolve to ‘bridge.mokka.drunkenmonk.org’ which resolves to ‘192.168.2.1’ according to /etc/hosts file. So dnsmasq will send ‘192.168.2.1’ as response.

Thats all from configuration stuff. These changes will take effect once you restart dnsmasq service. But this does not means that all your DNS queries will go through your machine’s dnsmasq daemon, If you have nameserver entry in /etc/resolv.conf, your machine’s DNS queries will go to /etc/resolv.conf nameservers. So, there is no use of using dnsmasq. You have to make sure you are not creating entries in /etc/resolv.conf. If you are a home user, make sure you select ‘Automatic (DHCP) Address only’ in ‘Method’ drop-down list inside ‘ipv4 settings’ tab in NetworkManger.

You can check that you are getting response from dhsmasq with below commands,

$ dig ANY drunkenmonk.org
; <> DiG 9.9.1-P2 <> ANY drunkenmonk.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39480
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;drunkenmonk.org.               IN      ANY
;; ANSWER SECTION:
drunkenmonk.org.        0       IN      CNAME   bridge.mokka.drunkenmonk.org.
bridge.mokka.drunkenmonk.org. 0 IN A    192.168.2.1
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Aug  6 00:15:16 2012
;; MSG SIZE  rcvd: 99

The line contains ‘SERVER:’ tell us that our DNS query is served by ‘127.0.0.1’, means our local machine. Here is the command to verify MX record,

$ dig MX drunkenmonk.org
; <> DiG 9.9.1-P2 <> MX drunkenmonk.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14301
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;drunkenmonk.org.               IN      MX
;; ANSWER SECTION:
drunkenmonk.org.        0       IN      MX      30 mail.drunkenmonk.org.
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Aug  6 00:16:38 2012
;; MSG SIZE  rcvd: 71

If you are not introduced to Linux’s bridge interface, you may wonder what the hell is ‘br0’. Well , it is a virtual device in Linux which act like a physical network device (like eth0). I have particularly configured my dnsmasq to serve DHCP requests to whoever connect through my bridge interface. Howto use this ‘br0’ effectively to configure networking in my Qemu virtual machines is going to be my next article. Have a nice day!!

GNU Gettext – Yet Another Tutorial

Well, developing a simple C program is easy, but developing it in a internationalized way (Yeah!! all those l10n, m17n and i18n thingy) is not so easy unless you understand Autotools. However, understanding it may take some time (atleast it took some time for me). In this post, I’m trying to explain how I learnt it. It may be wrong way, but atleast I can recollect what I did today in future.

Lets just create a simple C project. Obviously without any doubt, it should be called ‘helloworld’. Lets just create the directory tree first.

$ mkdir -p helloworld/{src,man}

Switch to ‘helloworld/src’ and create two files ‘helloworld.h’ and ‘helloworld.c’

/* helloworld/src/helloworld.h */
#ifndef __HELLOWORLD__
#define __HELLOWORLD__

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <libintl.h>
#include <locale.h>

#define _(STRING) gettext(STRING)

#endif

We have to include libintl.h to get ‘bindtextdomain(3)‘, ‘textdomain(3)‘ and ‘gettext(3)‘ functions. We have to include locale.h to get ‘setlocale(3)‘ function, let see why we need these functions,

setlocale()

Every glibc executable starts with the default locale called ‘C’. We use ‘setlocale()’ function to switch to different locale, this function takes two parameters ‘category’ and ‘locale’, ‘category’ indicates which locale variable we want to change and ‘locale’ contains what is the new value, If ‘locale’ is “”, setlocale() will get the value from the corresponding environment variable (see man page for more details).

bindtextdomain()

The gettext framework works as follows,

  • Get all the english output strings from the sources and generate a .pot file
  • Translate the english strings in the .pot file in different language and create .po file for each language
  • Generate .gmo binary files from the .po file for each language
  • Make the executable read corresponding translation from the .gmo file according to the locale settings each time it wants to print a message

To do the last step, we have to specify where the .gmo files are available. For that purpose, we use ‘bindtextdomain()’, it takes two arguments, ‘domainname’ and ‘dirname’. ‘domainname’ is the name we choose to group all our .gmo files under one place. Most of the time, we use the name of our project as ‘domainname’. ‘dirname’ is the common directory where different project’s .gmo files were placed. Usually it is ‘/usr/share/locale’.

textdomain()

we have to set the ‘textdomain’ so that executable will get the translated messages from the .gmo files correctly. ‘textdomain()’ takes only one argument ‘domainname’ which is the name of our project.

gettext()

Finally we have to wrap every output string to make them pass through ‘gettext()’ so that it can catch the correct translated string from .gmo files. We defined a macro ‘_()’ alias to ‘gettext()’ because we are lazy(aren’t we!?) to type ‘gettext()’ everytime.

So, here is the helloworld.c

/* helloworld/src/helloworld.c */
#include <helloworld.h>

int main(int argc, char *argv[])
{

  setlocale(LC_ALL, "");

#ifdef ENABLE_NLS
  bindtextdomain(PACKAGE, LOCALEDIR);
  textdomain(PACKAGE);
#endif

  printf(_("hello world\n"));

  return(0);
}

Now, we have to replace ‘PACKAGE’ and ‘LOCALEDIR’ macros to the real values. Here comes autotools, automake can give real value to ‘PACKAGE’ at compile time and automake also have a way to define LOCALEDIR at compile time, Lets do autotools by creating following files,

# helloworld/src/Makefile.am
bin_PROGRAMS = helloworld
helloworld_SOURCES = helloworld.c helloworld.h
DEFS += -DLOCALEDIR=\"$(localedir)\"
# helloworld/man/helloworld.1
helloworld :) !!! check after sometime
to see the real man page
# helloworld/man/Makefile.am
dist_man_MANS = helloworld.1
# helloworld/Makefile.am
SUBDIRS = src man

We need to run ‘autoscan’ to generate ‘configure.scan’ file. Rename ‘configure.scan’ to ‘configure.ac’ and edit that file according to the project’s need. I can’t explain all the autoconf macros within this blog post, see the end of this blog post to get the links for further reading.

$ cd helloworld
$ autoscan

Here is the customized ‘configure.ac’ file,

# helloworld/configure.ac
#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

AC_INIT([helloworld], [0.1], [mokka at comedysite dot com])
AC_CONFIG_SRCDIR([src/helloworld.c])

# Automake init
AM_INIT_AUTOMAKE([foreign -Wall])

# Checks for programs.
AC_PROG_CC
AM_PROG_CC_C_O

# Gettext init
AM_GNU_GETTEXT_VERSION([0.18])
AM_GNU_GETTEXT([external])

# Checks for libraries.

# Checks for header files.
AC_CHECK_HEADERS([libintl.h locale.h stdlib.h])

# Checks for typedefs, structures, and compiler characteristics.

# Checks for library functions.
AC_CHECK_FUNCS([setlocale])

AC_CONFIG_FILES([Makefile
                 man/Makefile
                 src/Makefile])
AC_OUTPUT

Now we have to run ‘gettextize’ under ‘helloworld’ directory to put gettext settings into ‘configure.ac’ and ‘Makefile.am’.

$ cd helloworld
$ gettextize

If things go well, you can see ‘helloworld/po’ directory and modifications into ‘configure.ac’ and ‘Makefile.am’. Now we can run ‘autoreconf’ to finish autotools procedure.

$ cd helloworld
$ autoreconf --force --install --verbose

Now, switch to ‘helloworld/po’ and rename ‘Makevars.template’ to ‘Makevars’. Inside ‘Makevars’ file, you may have to give inputs to some variables, may be atleast to ‘MSGID_BUGS_ADDRESS’, Here is a way to add your email address to that variable

$ cd helloworld/po
$ mv Makevars.template Makevars
$ sed -i '/^MSGID/s/$/mokka at comedytime dot com/g' Makevars

Now, we need to add the source filenames to ‘POTFILES.in’, Here a way,

$ cd helloworld/po
$ find ../src -name '*.c' -o -name '*.h' | sed 's/\.\.\///g' >> POTFILES.in

Time to compile,

$ cd helloworld
$ ./configure
$ make

You can see PACKAGE, LOCALEDIR macro definitions when make compile helloworld.c. As a programmer, your job is almost done.

Now switch yourself as a translator. go to ‘helloworld/po’ directory and generate a po file for your language using ‘msginit’, you have to provide ‘locale’ using -l option. You should know the languagecode and countrycode to construct ‘locale’ string. ‘msginit’ will ask for your email-id to put yourself into the translators list. Here I’m translating for ‘Tamil’ (ta_IN.utf8).

$ cd helloworld/po
$ msginit -i helloworld.pot -o ta.po -l ta_IN.utf8

I edited ta.po file with gedit+ibus, translated the word “hello world\n” to “வனக்கம்\n’. Now, I have to add my language to LINGUAS file. LINGUAS file contains languagecodes which have corresponding translated .po file inside ‘helloworld/po’ directory.

# helloworld/po/LINGUAS
ta

Now its time to generate binary .gmo file. Before that, We have to re-run ‘autoreconf’ to regenerate the helloworld/po/Makefile.in, because we updated LINGUAS file.

$ cd helloworld
$ make distclean
$ autoreconf --force --install --verbose
$ cd po
$ make update-gmo
rm -f ta.gmo && /usr/bin/gmsgfmt -c --statistics --verbose -o ta.gmo ta.po
ta.po: 1 translated message.
$

If your translation don’t have any errors, you will see ‘1 translated message’. Few more steps to achieve our goal, that is, creating distribution tarball and install our program to see the result.

$ cd helloworld
$ make distclean
$ make dist-bzip2
$ mkdir -p /tmp/buildir
$ mv helloworld-0.1.tar.bz2 /tmp/builddir
$ cd /tmp/builddir
$ tar xvjf helloworld-0.1.tar.bz2
$ cd helloworld-0.1
$ ./configure --prefix="/tmp/destdir"
$ make install
$ LANG="ta_IN.utf8" /tmp/destdir/bin/helloworld
வனக்கம்
$ /tmp/destdir/bin/helloworld
hello world
$

Thats it. My ‘helloworld’ program can say “வனக்கம்” now. You can also make it to speak your favourite language!!

References

There is another beautiful tutorial for gettext available at oriya.sarovar.org.

pcapstreamer – A packet dumper

Hi guys,

Happy Holidays!!

This year is about to finish, thinking about this year, lot of things happened, love, break-up, home, health, mother, father, work and more importently passion. Well, all is well and life is moving ahead. Still I’m travelling alone, in my own path. (What the hell am I, this is suppose to be technical post, shit!! crap philosophy!!)

I got some free time and spent that time learning libpcap. For those who don’t know, it is used in most of the network monitoring/capturing tools in *nix world. Very powerful.

The ‘tcpdump(1)‘ command is one such tool which uses libpcap (actually they are the one who created libpcap from tcpdump) to dump information about packets. It has a robest filtering mechanism to narrow down packet capturing to specific packets.

While trying to understand filter expressions in tcpdump, I got an Idea, I thought why not just convert the bytes in packets to strings and print them in stdout, this way, we can see the exact bytes, so further processing can be done my other unix tools (like awk, perl etc.,).

So, I just wrote a tool called ‘pcapstreamer‘ to capture packets from linux’s ‘any’ psudo-interface. Its very simple tool, you need to run this tool as root user. It just dump packets, thats all. Here is an example, this shows one packet dumped into stdout.

$ sudo ./pcapstreamer
[cl:76 l:76 t:20111226085033.641612] 00000000 00000000 00000011 00000100 00000000 00000110 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00001000 00000000 01000101 00000000 00000000 00111100 01000111 10001100 01000000 00000000 01000000 00000110 11110101 00101101 01111111 00000000 00000000 00000001 01111111 00000000 00000000 00000001 11100101 01100011 00010101 10110011 01111000 00011101 00100110 01010100 00000000 00000000 00000000 00000000 10100000 00000010 10000000 00011000 11111110 00110000 00000000 00000000 00000010 00000100 01000000 00001100 00000100 00000010 00001000 00001010 00000000 10100010 01100001 11011000 00000000 00000000 00000000 00000000 00000001 00000011 00000011 00000101

Here ‘cl:76’ and ‘l:76’ indicates ‘captured length’, ‘t:20111226085033.641612’ indicates ‘timestamp’ in localtime. Other strings are just pure raw packet.

Linux Cooked Header

To understand first 16 bytes, we need to understand ‘Linux Cooked Header‘. First 2 bytes “00000000 00000000’ or “0x00” represents that this is an incoming packet. To understand the next 2 bytes, we need to refer linux’s ARPHRD_. 3rd and 4th bytes “00000011 00000100” or “decimal 772” indicates that this packet is coming into loopback interface. 5th and 6th bytes “00000000 00000110” or “0x0006” indicates the length of link-level address, the next 8 bytes (7th byte to 14th byte) represents the link-level address, however we should take only the next 6 bytes as link-level address, two more bytes (13th and 14th) are padded with zero. 15th and 16th bytes “00000000 00001000” or “0x0008” represents ‘ethertype‘ as ‘ip’, this tells us that this is an ‘ip’ packet. This ends the link-level header (data-link layer in OSI). We are now moving to ‘ip’ header (network layer in OSI)

IP Header

To understand details from 17th byte to 36th byte, we need to refer IP Header. Higher order 4 bits in 17th byte “0100” or “0x4” indicates that this ip packet is an ipv4 packet. Lower order 4 bits in 17th byte “0101” or “0x5” indicates IHL (Internet Header Length) usually this defaults to 5. 18th bytes represents ‘differentiated services’ usually 0. 19th and 20th bytes “00000000 00111100” or “0x003c” or “Decimal 60” represents remaining bytes count (CaptureLength minus Linux-Cooked-Header length). 21st and 22nd bytes “01000111 10001100” indicates identification. Higher order 3 bits in 23rd and 24th bytes “010” indicates that this packet is not fragmented, remaining 13 bits indicates fragment offset. 25th byte “01000000” or “0x40” or “Decimal 64” indicates TTL value. 26th byte “00000110” or “0x06” indicates that this is a ‘tcp’ packet. 27th and 28th packets indicates Header Checksum. 29th to 32nd bytes indicates source ip address (127.0.0.1) and 33rd to 36th byte indicates destination ip address (127.0.0.1). This ends the ‘ip header’, we are now moving to ‘tcp’ header (Transport layer in OSI).

TCP Header

To understand details from 37th byte to 76th byte, we need to refer ‘TCP Header‘. 37th and 38th bytes “11100101 01100011” or “decimal 58723” indicates the source port number. 39th and 40th bytes “00010101 10110011” or “decimal 5555” indicates destination port number (means incoming packet is trying to connect port 5555). 41st to 44th byte indicates sequence number and 45th to 48th byte indicates sequence acknowledgement number. Higher order 4 bits in 49th and 50th byte “1010” or “Decimal 10” indicates Data offset, means there are 10*4=40 bytes in TCP header. Next 3 higher order bits are reserved in 49th byte 50th byte. Next 3 bits indicates ECN. Next 6 bits “000010” or “0x02” indicates that ‘SYN’ flag was set in Control bits. 51st and 52 bytes indicates window size, means the sender is willing to accept “10000000 00011000” or “decimal 32792” bytes in the response packet. 53rd and 54th bytes indicates checksum. 55th and 56th bytes indicates Urgent pointer, usually 0.

Options

Inside TCP header, bytes 57 to 76 contains value based on 50th byte(Data Offset). In this particular packet, 50th byte has (0xa), which means, TCP header in this packet contains totally 40bytes. Mandatory TCP fields (from 37th byte to 56th byte) are already discussed, but we have 20 more bytes to decode, these bytes are represented as ‘Options’ in TCP header. They may occur or they may not occur in a TCP packet. Mostly they occur in SYN packet.

Here, 57th byte (0x02) represents option-kind, 58th byte represents option-length (0x04). Both 57th and 58th bytes represents that 59th and 60th bytes “01000000 00001100” or “0x400c” or “Decimal 16396” indicates “Maximum Segment Size“. 61st byte (0x04) represents option-kind, 62nd byte (0x02) represents option-length, both bytes represents “SACK permitted“. 63rd byte (0x08) represents option-kind, 64th byte (0x0a) represents option-length, both bytes indicates that from 65th byte to 68th byte contains ‘TSVal‘ and from 69th byte to 72nd byte contains ‘TSecr‘. 73rd byte “0x01” indicates option-kind as ‘No-Operation‘, 74th byte (0x03) indicates option-kind, 75 byte indicates (0x03) option-length, both bytes indicates that 76th byte (0x05) contains ‘WSOpt (Window Scale Option)‘, which means, the host which sent this packet can accept upto “32792 * (2^5)” or (windowsize[byte51&52] * (2^wsopt[byte76])) before sending ACK.

pcapstreamer with awk

To display only ICMP packets, we can use the following commandline

$ sudo ./pcapstreamer 2>/dev/null | awk '{if($26 ~ "00000001"){print $0;}}'

To display only SYN packets, we can use the following commandline

$ sudo ./pcapstreamer 2>/dev/null | awk '{ctrlbytes=$49$50; if(ctrlbytes ~ "^.......000010...$"){print $0;}}'

I hope this utility may be useful for newbies like me to learn networking. Have a great new year.

Systemd for Simple Backup

Hi,

Nice to get back, Recently switched to F15. Wow!! my userland changed heavily. Previously, its very simple things like,

SysVinit for booting
Udev for devices
pppconfig for Network
Xfce for GUI
ALSA for sound
ffmpeg & mplayer for video

Now,

Systemd for booting, daemons
Udev, DBus, UDisks for devices
ModemManager+NetworkManager for Network
Gnome3 for GUI
PulseAudio+ALSA for sound
GStreamer+totem for video

These Technologies are very interesting to learn. I’m not going to explain about each. But, I went through systemd and came up with a solution for my backup problem.

Backup Problem:

While in its last phase, my previous laptop teached me the importance of external backup disks. So, I bought an external 512GB Segate, and backed-up /home/${HOME} tree. It helped me to quickly get back my files to F15. However, one problem is, maintaining my backup. I thought If I plug my external drive, someone should automatically copy all the new files resides in my current /home/${HOME} in F15 to that drive. There are lot of ways to do it. I can think of two main ways,

* Write an udev rule to call a script which will mount that external backup partition and rsync /home/${HOME} in F15 to that external partition. Pros: Fairly straight forward, udevrulefile+rsyncscript will do it. Cons: No control, this will copy every time udev detects that external drive. If there is lot of files to copy, then this will make a mess.

* Use systemd to call a script whenever that external backup partition gets mounted. Pros: You have full control, create a backup.service for systemd, create backup.bash to rsync /home/${HOME} to backup disk. Enable that service in systemd to automatically do rsync, or just load that service to systemd and only start that service If you want to backup your files. Cons: Need to pass one more layer to run the actual rsync script.

So, I took-up systemd

There are lot of ways you can trigger your service in systemd. Also, you can depend on another systemd unit to trigger your service. In this case, I depend on a mount unit to trigger my backup.service. This service, in-turn, will trigger backup.sh script.

To define a systemd unit, you need to know what type of unit you want to create. Currently there are 10 types of units systemd can understand [read systemd.unit(5)]. For backup job, I used two type units, one is systemd.mount(5) amd systemd.service(5). If you go through systemd.mount(5) you will understand that systemd will automatically load this units whenever systemd saw a block device. So, systemd will automatically provide ‘media-ExternalBackupPartition.mount’ whenever I insert my external hard disk . I only need to define the next unit, backup.service for systemd.

Note: There is a story behind the name ‘media-ExternalBackupPartition’. I’ll tell you at the end of this post. Lets just continue with systemd for now.

To define a unit for systemd, you need to create a file as /etc/systemd/system/name.unit; (for my backup problem, unit file is /etc/systemd/system/backup.service). Here ‘name’, may be anything relevent to your job and ‘unit’ must be one of ‘service’, ‘socket’, ‘device’, ‘mount’, ‘automount’, ‘swap’, ‘target’, ‘path’, ‘timer’ and ‘snapshot’. Read systemd.unit(5) man pages for more precise information.

Unit Class:
Unit definition files contains information in .ini format. One of the class ‘[Unit]’ must exist in every unit file. Here is the [Unit] class for backup.service

[Unit]
Description='sync my files with external backup drive'
Requires=media-ExternalBackupPartition.mount
After=media-ExternalBackupPartition.mount

Here ‘Description’ is a general description for your service. We need to put the required unit which will trigger this new unit in ‘Requires’ field. Systemd will run this unit once all the units in ‘Requires’ fields satisfied. We can call these ‘Requires’ units as parent units, and your unit as child. However, systemd will not wait for parent units to complete to run child unit. We need to explicitly ask systemd to wait for parent units to complete, For this purpose we have ‘After’ field. If you define which parent unit needs to be completed before your child unit could run, you need to mention it in ‘After’ field.

For my backup.service child unit, ‘media-ExternalBackupPartition.mount’ unit must be satisfied in systemd. That means, my external HD partition must be mounted inside ‘/media/ExternalBackupPartition’ path. Also, Using ‘After=’ field, I instructed systemd, not to start this child unit before ‘mount-ExternalBackupPartition.mount’ finishes.

Service Class:

Now we need to define what to do once the requirements satisfies. For that purpose, we need to define ‘[Service]’ class, Here is the service class for backup.service,

[Service]
Type=simple
ExecStart=/home/mohan/Development/scripts/backup.sh

This ‘[Service]’ class is specific to ‘systemd.service’ units. Here, ‘ExecStart=/home/mohan/Development/scripts/backup.sh’ asks systemd to run ‘backup.sh’ whenever ‘backup.service’ satisfies. You can do lot of customization to setup the execution environment before start running any commands, such as log redirection, demonizing etc., there are lot of fields to use in ‘[Service]’ class, but I simply used ‘Type=simple’ to tell systemd, that no need to do any change in execution environment. The script ‘backup.sh’ will take care of all the redirection within itself.

Install Class:
In SysV init system, we use ‘chkconfig (redhat/fedora)’ or ‘update-rc.d (debian)’ to enable or disable a service. In systemd, we use this ‘[Install]’ class to enable or disable our ‘backup.service’ unit so that it will work even after a restart. Here is install class for backup.service,

[Install]
WantedBy=media-ExternalBackupPartition.mount

In systemd, enabling a service means, adding a symlink to ‘/etc/systemd/system/name.service.wants/’ directory. disabling a service means, removing that symlink. ‘systemctl’ command can do this add/remove symlink automatically when we call it with ‘systemctl enable backup.service’ or ‘systemctl disable backup.service’, but we need to say the parent unit name, thats why we have this ‘[Install]’ class. Simply, we need to mention that parent unit in ‘WantedBy=’ field.

Execution:

Once the unit files are ready, we need to enable them into systemd. For backup.service, I executed following commands to setup the service

$ sudo cp ~/backup.service /etc/systemd/system/backup.service
$ sudo systemctl daemon-reload

I just copied backup.service unit file to systemd’s location and asked systemd to reload unit definitions. Now we can check if things loaded properly or not using following command,

$ sudo systemctl status backup.service
backup.service - 'sync my files with external backup drive'
          Loaded: loaded (/etc/systemd/system/backup.service)
          Active: inactive (dead)
          CGroup: name=systemd:/system/backup.service
$

systemd will say ‘Loaded : error’ if it can’t understand any defnintion in backup.service or if it can’t satisfy the definitions. Otherwise we can start this service using following command, we need to make sure the final rsync script ‘backup.sh’ exists in the location pointed by ‘ExecStart=’ field.

$ sudo systemctl start backup.service

This will start syncing new files to External backup HD drive, only when It is plugged-in and mounted. Otherwise, the service will fail. you can check the status again using ‘systemctl status’. Once you checked that the service is working as intended, we can enable this service (I mean, creating symlinks) using following command,

$ sudo systemctl enable backup.service

We can verify the symlink as below to make sure parent-child linking is done correctly.

$ ls -l /etc/systemd/system/media-ExternalBackupPartition.mount.wants/baskup.service
lrwxrwxrwx 1 root root 34 Nov  9 02:08 /etc/systemd/system/media-ExternalBackupPartition.mount.wants/backup.service -> /etc/systemd/system/backup.service

If we don’t want to copy automatically, we can disable it using below command,

$ sudo systemctl disable backup.service

Even If the service is disabled, you can start/stop the service. Systemd will recognize the backup.service, check it’s dependencies and execute ‘backup.sh’ correctly.

media-ExternalBackupPartition.mount:

As I already said, systemd will automatically create units using udev, so when my external HD plugs-in, udev will tell to udisks that a new partition is available, then udisks will mount that partition inside ‘/media/uuid’ location, then systemd will create a unit as ‘media-uuid.mount'(systemd uses ‘-‘ instead of ‘/’ for path seperation). But specifying ‘Requires=media-uuid.mount’ inside backup.service file is not working. Thus, I used a simple udev rule to rename udisk’s mount path, here is the rule file,

$ cat 99-rename-udisk-mountpoint.rules
ENV{ID_FS_UUID}=="251c683d-bce0-489c-aab5-f684a9a1f3b2",ENV{ID_FS_UUID}="ExternalBackupPartition"
$ sudo cp ~/99-rename-udisk-mountpoint.rules /etc/udev/rules.d

This above two commands, will modify udisk’s mount path to ‘/media/ExternalBackupPartition’ instead of ‘/media/251c683d-bce0-489c-aab5-f684a9a1f3b2′, thus systemd will automatically create ‘media-ExternalBackupPartition.mount’ instead of ‘media-251c683d-bce0-489c-aab5-f684a9a1f3b2.mount’

Finally, Thanks for reaching this line. I hope this long boring article will help you to understand something about systemd.

Creating Fedora DVD Repo

Hi,

Its almost 1 year since my last post, what happens to me? It a 1 year story to say. But in short, my laptop became older and eventually went down. Graphics card gone. It took a while for me to buy another new one. Here I am.

In between lot thing happened, Something I just wanted to post, but now I can’t remember them. As a fresh start, This is what happened when I got my new ‘Samsung Notebook’. Ordered through FlipKart and got it by last week.

I know that I can’t get Internet because I cancelled my netconnect+ subscription way back when my previous lappy started giving troubles. Thus, My beloved debian is out of reach. So, I decided to get Fedora or Mint DVD from someone and Install it. Thanks to Srini, Amachu and his Gang, for providing Fedora DVD to me. Installation went smoothly and for the first time, I’m in Gnome3,

Fedora 15 Desktop
Fedora 15 Desktop

Installation went smoothly and to my surprise, no more tinkering on mobile modem. Thanks to those guys who created profile for Aircel. At first, bluetooth daemon didn’t start, googled and fixed it.

While playing with Gnome3, I thought of using installation DVD as my repo, because I have limited network with very little speed, It would be better to use existing DVD as my repo. So, that starts my hunger and this is how I ended.

1. Inserted my DVD and monitored how my DVD is mounting. Seems ‘org.freedesktop.UDisks’ takes care of mounting and it always mount my DVD inside ‘/media’ folder taking mount directory name from ‘ENV{ID_FS_LABEL_ENC}” Udev environment variable. I came to know that yum is facing some trouble If I have a ‘fedora-dvd.repo’ file like below under ‘/etc/yum/yum.repos.d’ directory,

[fedora-dvd]
name=Fedora DVD $releasever - $basearch
baseurl=file:///media/Fedora 15 i386 DVD
enabled=1
gpgcheck=1
gpgkey=file:///media/Fedora 15 i386 DVD/RPM-GPG-KEY-fedora-$basearch

So, it seems yum cannot handle white-space in URI. Then, I decided to write a Udev rule to rename ‘Fedora 15 i386 DVD’ to ‘Fedora15i386DVD’, So that ‘org.freedesktop.UDisks’ will mount it as ‘file:///media/Fedora15i386DVD’. Here is the rule file

$ cat /etc/udev/rules.d/99-rename-fedora-dvd.rules
ENV{ID_FS_LABEL_ENC}=="Fedora\x2015\x20i386\x20DVD", ENV{ID_FS_LABEL_ENC}="Fedora15i386DVD"
$

Then, I recreated fedora-dvd.repo as below

$ cat /etc/yum.repos.d/fedora-dvd.repo
[fedora-dvd]
name=Fedora DVD $releasever - $basearch
baseurl=file:///media/Fedora15i386DVD
enabled=1
gpgcheck=1
gpgkey=file:///media/Fedora15i386DVD/RPM-GPG-KEY-fedora-$basearch
$

One of the main reason to add this ‘Fedora15i386DVD’ repo is to install development tools without internet. In a way it is more useful than a plain LiveCD.

Don’t forget to insert Fedora DVD If you follow this setup and want to install something from Fedora DVD. A little google tells me that there are lot of ways to achieve this DVD-to-Repo thing in Fedora, I’m relatively new to Fedora, I will learn as I move on.

Howto bake Persistent Live USB – Debian Way (live-build)

Hey guys!! wish you happy festive days!!

For long time I was thinking about creating my own system. Recently, during my experiment with ‘mic2’, I came to know about ‘live-build’ package.

‘mic2’ is a package which contains tools to create Meego images. I’m yet to create my Meego Image. During Meego image creation, I had lot to do with my USB pendrive. My pendrive previously had puppy for system rescue purpose. After spending much time with Meego without any success, I thought of creating my rescue USB by downloading puppy. But decided to give ‘live-build’ a try.

First we need to install ‘live-build’

sudo apt-get install live-build cdebootstrap

‘live-build’ is a set of scripts to build customized debian iso/usb-hdd images. I’m not going to explain each scripts, instead, just lets start building a Debian Persistant Live USB with ‘rescue’ packages from ‘Testing’ repository for ‘i386’ architecture. My host system is ‘Debian Testing (amd64)’ Here is the steps,

Create a directory and go inside

mkdir live-build-base && cd live-build-base

Execute below command

sudo lb config --architecture i386 --binary-images usb-hdd --binary-filesystem fat32 --bootappend-live 'persistent' --bootstrap cdebootstrap --distribution testing --mirror-binary http://ftp.de.debian.org/debian/ --packages 'usb-modeswitch' --packages-lists 'rescue'

This tells ‘–architecture’ for the live system should be ‘i386’, you can give any of debian ports like ‘amd64’, ‘sparc’, ‘powerpc’, ‘armel’, ‘hurd-i386’, ‘kfreebsd-gnu’ etc., to this option. But careful, we are not using qemu to build the image. we are using cdebootstrap, thus, an ‘amd64’ host system can bake ‘amd64’, ‘i386’, ‘hurd-i386’, ‘kfreebsd-gnu’ images, but it cannot bake ‘armel’, ‘sparc’ or ‘powerpc’.

The next option ‘–binary-images’ tells what kind of image to create, like ‘iso’ or ‘iso-hybrid’ or ‘usb-hdd’ etc., (see ‘man lb_config’). Next option ‘–binary-filesystem’ tells the format of the final root filesystem.

Next one ‘–bootappend-live’ passes options to ‘live-initramfs’ which is the main tool to create live debian system, see ‘man live-initramfs’ to know about its options. ‘persistent’ informs ‘live-initramfs’ to generate ‘initrd’ image which will search for partitions with label ‘live-rw’ or ‘home-rw’ at boot time, if ‘initrd’ finds, it will instruct ‘aufs’ to stage that partition as a rw layer for root filesystem.

‘–bootstrap cdebootstrap’ will use ‘cdebootstrap’ while creating the image. you should install ‘cdebootstrap’ in your host system first.

‘–distribution testing’ indicates which repo to download packages, you can give ‘stable’, ‘unstable’, ‘testing’, ‘experimental’, ‘lenny’, ‘squeeze’, ‘sid’ and ‘wheezy’.

‘–mirror-binary’ points to the repo URL

‘–packages usb-modeswitch’ will additionally install ‘usb-modeswitch’ package. I need this, because I’m using a netconnect datacard for internet access.

‘–packages-list rescue’ will get packages from /usr/share/live/build/lists/rescue file. This file is preconfigured to install specific packages for specific purpose, we can give ‘rescue’, ‘gnome’, ‘kde’, ‘gnustep’, ‘lxde’, ‘studio’ etc., each have a corresponding file inside /usr/share/live/build/lists directory in your host system.

Above options prepares ‘live-build-base’ directory to build your customized debian. Now, its time to build, execute the below command,

sudo lb build

Once started, ‘lb build’ will take some time to bake the required image. Finally, you will get ‘build.img’ with all your customized packages installed. Next step should be dd-ing the image to your USB pendrive. Insert it, and execute the following command,

sudo dd if=build.img of=/dev/sdX bs=4096

Here, ‘/dev/sdX’ indicates your loaded pendrive name, it may be ‘/dev/sdb’ or ‘/dev/sdc’. Once ‘dd’ finishes, reboot your system and boot from your USB pendrive. You will be inside ‘Debian Live’ !!!!

Next, we need to create ‘live-rw’ partition in your pendrive for persistent storage (means automatically saving the changes you do in your live system). Just create a partition with label as ‘live-rw’ from the free space left in your pendrive using ‘gparted’ or ‘fdisk’. Then boot your machine from USB drive, once you are inside ‘Debian Live’, you can see the new partition mounded as ‘aufs’ rw layer for root filesystem in /live/cow.

During ‘lb build’ phase, if any error occurs, or you decide to change your configuration, remove all directories except ‘cache’ from ‘live-build-base’ directory. Then remove ‘.stage’ directory and start ‘lb config’ with your options again from ‘live-build-base’ directory. This will reduce package download time, but if you are building for different architecture, then its better to delete ‘cache’ also. Here is the screenshot of my image running under QEMU,

Thats it!! hope you will bake a better ‘Debian Live cake’ than me!!!