Lyte's Blog

Bad code, bad humour and bad hair.

Oops I Needed That Partition Table…

I just lost the partition table on a drive with a non-standard layout and managed to get the data back… phew.

I originally had something like:

1
2
3
4
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          26      208813+  83  Linux
/dev/sda2              27        3943    31463302+  8e  Linux LVM
/dev/sda3            3944        5221    10265535    7  HPFS/NTFS

Note: partition table above is from a VM I spun up to write this entry.

That is to say I had a drive dual booting Linux and Windows, with Windows at the end of drive (because Russel sort of implied it was better).

The person who owns the drive in question wanted all the space claimed by Linux reallocated back to Windows.

After confirming backups of critical documents I fired up gparted in a live CD… which all seems swimmingly ok until the Laptop powered off (because the power plug had dropped out and the battery went flat).

After trying to boot up I discovered a corrupted partition table.

This would probably be ok (given the backups of documents) if not for the fact that I had no appropriate Windows installation media and the owner of the Laptop couldn’t find the original installation disks.

Oh noes! (I scream loudly inside my head so as not to alarm the owner of the Laptop)

After booting back in to the live CD I have a partition table that looks more like this:

1
2
3
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          26      208813+  83  Linux
/dev/sda3              27        1304    10265535    7  HPFS/NTFS

“Ok” I hear you saying “just rewrite the partition table with the output you saved from “fdisk -l” before starting. “Oh you didn’t save the output before starting…”

After a bit of hair pulling (and googling) I come across gpart.

It found the old partition without breaking a sweat:

1
2
3
4
5
6
7
8
9
10
11
12
13
root@ubuntu:~# gpart /dev/sda

Begin scan...
Possible partition(Linux ext2), size(203mb), offset(0mb)
Possible partition(Windows NT/W2K FS), size(10024mb), offset(203mb)
Possible partition(Windows NT/W2K FS), size(10024mb), offset(30929mb)
End scan.
[...]
Primary partition(3)
   type: 007(0x07)(OS/2 HPFS, NTFS, QNX or Advanced UNIX)
   size: 10024mb #s(20531070) s(63344295-83875364)
   chs:  (1023/254/63)-(1023/254/63)d (3943/0/1)-(5220/254/63)r
[...]

After talking to fdisk nicely I was able to convince it to create the partition table that gpart was suggesting and subsequently able to mount, resize properly and validate the partition with Windows chkdsk tool… phew, crisis averted.

Next time I’m dding the disk to a USB drive first, no matter how long it takes.

Attack of the Killer Line Endings

If you save a bash script on Windows or email it as a plain text attachment through most Windows email clients, your script will probably stop working.

Why?

Hah! I’m not going to tell you yet, first try it out.

Download the killer_newlines script somewhere and confirm it’s not going to do anything evil:

1
2
3
4
$ curl -s http://lyte.id.au/bash/killer_newlines > killer_newlines
$ cat killer_newlines
#!/bin/bash
echo Hello World

Looks good, it doesn’t do anything dangerous.

Lets run it to make sure it still works:

1
2
3
4
5
6
$ chmod +x killer_newlines
$ ./killer_newlines
bash: ./killer_newlines: /bin/bash^M: bad interpreter: No such file or directory
$ bash killer_newlines
Hello World
: command not found: line 3:

Huh, what just happened?

Unix is just using LF (Line feed, ‘\n’, 0x0A, 10 in decimal) to signify the end of a line, where as Window (and DOS) also use CR (Carriage return, ‘\r’, 0x0D, 13 in decimal). See: Wikipedia’s Newline article for more info.

Bash only copes with Unix style line endings and we’ve introduced non Unix line endings.

But what specifically is happening, why is the error so bizarre?

Bash is trying to execute a command ‘\r’. Given there’s (usually) not a ‘\r’ executable in $PATH it errors.

You may also want to play around with these:

1
2
3
4
5
6
7
$ bash <(printf 'echo Hello World')
Hello World
$ bash <(printf 'echo Hello World\n')
Hello World
$ bash <(printf 'echo Hello World\n\r')
Hello World
: command not found

Weird huh?

Stop Typing Www.

It’s hard to pronounce.

It’s an out dated habit.

The first website didn’t have a “www.” prefix.

You have to type 4 extra characters each time you use it.

Websites that render pages both with and without a “www.” prefix have cache coherency problems.

Stop it, stop it now.

From now on http://www.lyte.id.au/* will redirect to this post, just to spite you.

Have a nice day.

Checking for Apt Security Updates With Nagios

On Ubuntu the update-notifier-common package provides a simplistic API to ask if there are security updates available.

I’ve written a little script to convert the output so that I can monitor multiple machines using Nagios:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#!/bin/bash

# Munge output of apt_check.py suitably for Nagios
#
# @author David Schoen - http://lyte.id.au/

# apt_check.py outputs <total updates as int>;<security updates as int> to stderr
# we take this, redirect it to stdin and then read in to local variables
IFS=';' read -r total security < <(/usr/lib/update-notifier/apt_check.py 2>&1)

if [[ $security -eq 0 ]]; then
  echo "APT OK - $security security, $total total updates"
else
  echo "APT WARNING - $security security, $total total updates"
  exit 1
fi

Place this somewhere Nagios (or NRPE) can execute it and call it like you would any other check command.

Remembering Long Running Commands

At least once a day I’ll start a command that runs for a few hours.

Usually I attach a sendmail command to the end of it so that I’ll know when it completes like so:

1
2
3
$ some really slow command; \
echo "Subject: some really slow command has completed" \
| sendmail me@exmaple.com

… but sometimes I just forget.

On Solaris I previously fired up a second screen session at this point and wrote something like:

1
2
3
$ pwait <pid of some really slow command>; \
echo "Subject: some really slow command has completed" \
| sendmail me@exmaple.com

but in Linux I haven’t been able to achieve the same thing:

1
2
$ wait 1234
bash: wait: pid 1234 is not a child of this shell

so I’ve been looking for an alternative solution.

I finally figured it out.

Press ctrl+z to pause the process:

1
2
$ some really slow command
[1]+  Stopped                 some really slow command

then simply fore-ground the process with the usual email alert appended on the end:

1
2
3
$ fg; \
echo "Subject: some really slow command has completed" \
| sendmail me@exmaple.com

Tracking Yourself in Apache Logs

Sometimes I want to filter out all log lines that aren’t generated by me when I’m debugging a problem with web servers.

At some point I worked out this trick and I thought I’d share it for others.

In Firefox it’s relatively trivial to add a unique string to your useragent. Fire up the about:config interface and add a new key like general.useragent.extra.yourname with a unique value: about:config

Then you can simply grep for your unique string in access logs:

1
2
root@foo:/var/log/apache2# tail -f access.log | grep 'lyte - asdf123123'
bar.example.com:80 127.0.0.1 - - [11/Mar/2011:01:48:06 +1100] "GET / HTTP/1.1" 200 497 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110221 Ubuntu/10.04 (lucid) Firefox/3.6.14 lyte - asdf123123"

Check Your ~/.xsession-errors File

Recently my backup drive started filling up really quickly

1
2
3
4
5
6
7
8
9
10
11
12
13
14
1.4M 2011-01-03_033006/home/xxxx/.xsession-errors
1.4M 2011-01-04_033006/home/xxxx/.xsession-errors
1.5M 2011-01-05_033006/home/xxxx/.xsession-errors
 16K 2011-01-06_033005/home/xxxx/.xsession-errors
8.0K 2011-01-07_072908/home/xxxx/.xsession-errors
 72K 2011-01-08_094527/home/xxxx/.xsession-errors
228M 2011-01-09_033006/home/xxxx/.xsession-errors
3.4G 2011-01-10_213703/home/xxxx/.xsession-errors
6.2G 2011-01-11_033006/home/xxxx/.xsession-errors
10.4G 2011-01-12_033006/home/xxxx/.xsession-errors
12.3G 2011-01-13_033006/home/xxxx/.xsession-errors
15.1G 2011-01-14_033006/home/xxxx/.xsession-errors
...
27.5G 2011-01-21_033005/home/xxxx/.xsession-errors

… you get the point.

Anyway because I’m storing every file that’s modified during the day again and again every night, my backups were growing by at minimum the size of the .xsession-errors log, which is suddenly HUGE.

I should probably look in to why that file is suddenly growing ultra quickly, but for now I think I’ll just exclude it from my backup.

Export All Tomboy Notes

Recently I’ve been trying Tomboy, but I’m giving up.

I haven’t found a good method of exporting all notes at once either built in or via a readily available plugin. You can import linked notes and even follow them recursively, but some of my notes don’t have direct links as they are intentionally only locatable via search.

To make the export work better I ended up creating a note that created a single link to every other note using xpath under bash.

First, move to the notes directory:

1
cd ~/.local/share/tomboy

Then, for every note grab the title using xpath:

1
2
3
4
for i in *.note; do
    xpath -q -e '/note/title' $i;
done \
| sed -r 's%^<title>(.*)</title>%\1%g'

Note: The sed command is horrible hack because I couldn’t find a graceful way to make xpath give me the value of the node without printing the actual tags that are matched. I was on the edge of writing a full blow script, but sed was good enough.

Copying the output of that in to a new note automatically created the links and exporting that note gave me HTML output that was much easier to move around.

IE - File Can Not Be Written to Cache

When connecting to an SSL site that is sending “Pragma: no-cache” headers it looks like all versions of IE fail to download files such as .doc, .pdf and .xls due to it refusing to let them pass through disk cache to be passed on the relevant application.

Some sites seem to suggest this is trivial to fix by adjusting the “Do not save encrypted pages to disk” setting in IE. In IE 8 this setting made no difference for me and in most cases the users I deal with that are using IE are unable to change such settings as they are locked down in Mandatory profiles or other forms of AD config.

See: http://answers.google.com/answers/threadview/id/142007.html

My hack to solve this problem is to strip out the Pragma header when the user agent matches MSIE at the Apache level:

1
2
3
# file downloads do not function in IE if "Pragma: no-cache" is set in the headers
BrowserMatch MSIE disable_pragma
Header unset Pragma env=disable_pragma

In this case it’s ok because we are also sending sensible Cache-Control headers which all HTTP/1.1 compliant software should be using anyway. As I know about everything between Apache and where SSL is terminated and I can ensure that Cache-Control is being sent by the program up stream from Apache, disabling Pragma shouldn’t really affect anything, but just in case we only disable it for IE. See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9

It would probably be a good idea to also run a test to ensure SSL is actually on, but I’m lazy and in this case it’s not possible as SSL is handled at a different layer that doesn’t let Apache know what’s going on.

Ssh Crackers Suck

Every now and then I rebuild a machine that I’m leaving permanently plugged in to the internet and available via ssh. I limit ssh to key based logins only and I encrypt all my keys, so I’m reasonably comfortable that this is “secure enough”, but after a couple of months I always notice that I’m getting heaps of connections to ssh that are slowing down my link.

The security implications don’t really bother me because I basically trust the rest of the security I’ve implemented but there is a trivial way to stop this and I always forget what it is.

I found it again so I thought I’d write it down somewhere… In Ubuntu, just install “denyhosts”. If your config is still mostly standard, sshd will be pumping authentication errors in to /var/log/auth.log and denyhosts will simply pick these up and put the relevant bad IPs in /etc/hosts.deny.

denyhosts picked up all but the bottom 3 IPs I had trying to crack my system:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# grep 'Invalid user' /var/log/auth.log | sed 's/.*Invalid user //g' | sed 's/.* from//g' | sort | uniq -c | sort -nr
   1404  203.110.240.71
    264  210.169.213.211
    179  210.51.191.165
    134  217.91.230.179
     75  222.73.205.9
     28  125.141.237.100
     14  58.254.201.113
      9  88.134.254.37
      9  219.238.166.101
      8  212.76.68.158
      2  217.97.185.35
      2  122.155.5.132
      2  118.129.153.43

… those guys weren’t very dedicated any way, I’ll give them another chance to lift their game.

They tried a massive range of different user names:

1
2
# grep 'Invalid user' /var/log/auth.log | sed 's/.*Invalid user //g' | sed 's/ from.*//g' | sort | uniq -c | sort -nr | wc -l
1071

Apparently the first few times they tried some of these names they had the wrong password though:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# grep 'Invalid user' /var/log/auth.log | sed 's/.*Invalid user //g' | sed 's/ from.*//g' | sort | uniq -c | sort -nr | head -20
     67 test
     49 oracle
     47 admin
     46 user
     37 guest
     24 postgres
     22 administrator
     21 web
     21 vmail
     21 master
     21 info
     19 teamspeak
     18 tomcat
     18 ftp
     17 webmaster
     15 jboss
     14 ubuntu
     14 operator
     14 eric
     13 vsifax

little did they know that password authentication is not configured to let anyone in, let alone this “test” fellow.