No? Try this, from an empty directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Set umask to 0002 (i.e don't mask out user/group bits)
$ umask 0002
# create a new git repo in the current dir
$ git init .
# create a new file
$ touch foo
# get the octal perms on it
$ stat -c "0%a %n" foo
0664 foo
# add and commit
$ git add foo
$ git commit -m "bar"
[master (root-commit) 51dc083] bar
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 foo

So I just added a file to git that has octal perms of 0664 but git reported 100644 – it’s safe to ignore the prefix, but why has the second last digit changed?

Here’s something else to try:

1
2
3
4
5
6
7
8

# Create a file with every (practical) octal mode
$ for file in 0{4..7}{0..7}{0..7}; do touch $file; chmod $file $file; done
# Stage those files
$ git add 0???
# Look at what perms git has staged:
$ git diff --staged | grep '^new file mode ' | sort | uniq -c
    128 new file mode 100644
    128 new file mode 100755

So git took all 256 different combinations of modes we gave it and stored 100644 half the time and 100755 the other half.

This is because git is really only looking at the u+x (user execute aka & 0100) bit – it doesn’t care about the rest of the perms.

So maybe I can give a better explanation.

First question – where is the spec for X-Cache? Well, there isn’t one – that little X- prefix indicates the header is not part of the spec, so its meaning may vary between proxy implementations (in fact with Varnish it’s common to add it yourself in the config file, so it could mean anything at all).

Why am I seeing these headers?

So with that out of the way I’ll focus on where I think you’re most likely to see them, Squid. Squid is a fairly common caching proxy, both as a forward (outbound) caching proxy and as a reverse (inbound/application accelerator/aggregating) proxy. Squid’s doco also fails to clearly define what the headers do, so if you’ve got here because you’re trying to figure out what they mean, there’s a good chance you’ve got a Squid proxy in the mix (even if you didn’t realise it).

What they mean

So, with Squid what these headers mean is:

• X-Cache: Did this proxy serve the result from cache (HIT for yes, MISS for no).
• X-Cache-Lookup: Did the proxy have a cacheable response to the request (HIT for yes, MISS for no).

This means if:

• both are HITs, you made a cacheable request and the proxy had a cacheable response that matched and it handed it back to you.
• X-Cache is a MISS and X-Cache-Lookup is a HIT, you made a request that had a cacheable response but you’ve forced a cache bypass – usually achieved by a hard refresh, commonly activated with Ctrl+F5 or by sending the headers Pragma: no-cache (HTTP/1.0) or Cache-Control: no-cache (HTTP/1.1).
• both are MISSes, you made a request (it doesn’t matter if it was cacheable) and there was no corresponding response object in the proxy cache.

This is completely irrelevant to browser cache – if you’re viewing browser cache and inspecting the headers in Chrome or Firebug then you’ll see what the status of the proxy was at the time the proxy returned it to your browser. Sorry if this is obvious, but a surprising number of people seemed to think that the browser cares and bothers to modify these headers, it doesn’t. Really, it doesn’t. I promise.

How you can test this for yourself

First, if you’re trying to use a browser to inspect headers, understand what it’s really saying, e.g in Google Chrome, look for the (from cache) next to the Status Code: section in the headers:

This means nothing has gone over the network and it brought the object out of browser cache.

All other browsers are left as an exercise for the reader as frankly it’s not the right tool for the job (I just figured I had to cover one, as everyone seems to do it this way anyway).

Fire up a bash terminal and get used to curl.

This is what I usually run:

1
2
3
4

$ curl -sv http://example.com/ 2>&1 > /dev/null | egrep '< (X-Cache|Age)'
< Age: 1
< X-Cache: HIT from example.com
< X-Cache-Lookup: HIT from example.com:80

There’s a lot going on here, so lets break it down:

• curl: this is the main thing we’re running,
• -sv: this enables both --silent and --verbose, which (counterintuitive I know) is the simplest way to get curl in to a mode where it dumps both the body and headers out in a useful format (yes I know you can use --head, but that changes the request format and invalidates enough testing to make it useless).
• http://example.com/: our URL of choice.
• 2>&1: Take STDERR (headers) and redirect it to STDOUT (so we can pipe it to egrep).
• > /dev/null: Ignore the original STDOUT (body).
• | egrep: pipe the headers through egrep.
• '< (X-Cache|Age): grab just headers starting with X-Cache or Age.

You may note I snuck that Age header you’ve been ignoring in there, you can thank me later.

So in this first example we’ve made a cacheable request (absent of Pragma: no-cache or similar headers) and the proxy has had a response from its upstream 1 second ago (Age: 1) that was cacheable.

Now that we have a basic command, lets fiddle with it to see what happens.

If we request the same thing, we should see the Age header count up by roughly the number of seconds since the start of either request:

1
2
3
4

$ curl -sv http://example.com/ 2>&1 > /dev/null | egrep '< (X-Cache|Age)'
< Age: 5
< X-Cache: HIT from example.com
< X-Cache-Lookup: HIT from example.com:80

If we request the same thing but with a hard refresh header (note: Squid and probably every other caching proxy ever can be configured to ignore these headers on the request side, so if you get different results here, 9 times out of 10 that’s why), we can see that the proxy had the object in cache, but didn’t use it:

1
2
3

$ curl -sv --header "Pragma: no-cache" http://example.com/ 2>&1 > /dev/null | egrep '< (X-Cache|Age)'
< X-Cache: MISS from example.com
< X-Cache-Lookup: HIT from example.com:80

If we request something we don’t expect the proxy to have seen before, the proxy neither has it in cache or serves it from cache:

1
2
3

$ curl -sv http://example.com/?"$(date -Ins)" 2>&1 > /dev/null | egrep '< (X-Cache|Age)'
< X-Cache: MISS from example.com
< X-Cache-Lookup: MISS from example.com:80

That said, this one particulary annoyed me.

PHP caches stat information. Why? I dunno really, VFS caches stat information too, you’d think it’d do a better job being that people other than PHP core developers designed it (so maybe some of them weren’t drunk at the time), plus it’s actually the right layer, it can see not only file system modifications made by the PHP you’re currently writing, but also all that other PHP you’ve written and the stuff Mr Jones wrote, also that strange bit of code that’s not PHP. So it would actually be able to cache the right things at VFS, instead PHP core devs actually thought it made sense to cache it in a PHP process, but what the hell lets leave OS problems to application developers and just move on.

So back on topic, PHP caches stat information or to put it another way, PHP has a stat cache (do you see what I did there?). Stat information are all those boring statistics you get about files when you write stat <file>, but really it’s so much more. It’s most of the data about a file that isn’t actually the file itself.

Great, so PHP caches stat information. That must make things faster or something? Possibly, but it also just makes it a massive hot squishy steaming pile of lies. Because any interaction with a file’s stat info is cached, if a file is modified in some way outside of PHP that you care about you now have to constantly call clearstatcache() so that when you check anything about the file you can be confident that PHP isn’t lying to you.

Did I mention PHP has a stat cache? A-packed-full-of-lies-but-at-least-it’s-faster-than-telling-you-the-truth-except-now-you-have-to-slow-things-down-by-constantly-clearing-it-stat-cache.

Somewhere about here you learn to deal with it, you think to yourself “it’s ok, I’ll just call clearstatcache() everywhere, it’s fine, it’s just a stupid cache that I can’t turn off, so I may as well deal with it, maybe I’ll write a Vim macro to insert clearstatcache() calls in front of any file system operations, that’ll fix it, it’ll be almost like PHP is a real language”.

Then, because you haven’t written that Vim macro (it’s a stupid idea any way) and you haven’t managed to turn off the broken stat cache (because you can’t) you run in to what must surely be a bug – even PHP core devs couldn’t consider this a feature, surely? You change ownership of a file using the built in chown() function, it’s a glorious thing, at first it’s owned by one user, then it’s owned by another user, cheers go up, standing ovations, it’s possible something built in to PHP actually works as designed. Cool. You go to check this using a PHP function (because like all PHP devs you totally do TDD and as such need to check such things) and because you updated the ownership information using a PHP function obviously the cache that PHP manages should be marked dirty and you’ll get back an accurate result right?

No.

What?

No, you don’t.

I’m going to say it again… What?

Ok it must be a bug.

Oh, no, you see, actually, it’s the first fscking example on the manual page.

Hmmmm….

Did I mention, PHP has a stat cache?

1
2
3
4
5
6

class IAmASpy {
  public $invocations = array();
  public function foo() {
      $this->invocations []= 'foo';
  }
}

or trying to use execution checks on mock objects to determine that things were called with the right arguments:

1
2
3
4

$mock = $this->GetMock('Foo');
$mock->expects($this->once())
    ->method('bar')
    ->with($this->identicalTo('baz'));

What A Pain!

What if you want to check the arguments going in to the last call? Well you can use at():

1

$mock->expects($this->at(7)) // ...

… better hope we never add any other calls!

What if we don’t know the exact parameter that it’s being called with and want to check it with something more complex? Well if you dig really hard in the manual you’ll find there’s a whole bunch of assertions that let you feed in crazier stuff like:

1
2
3
4

// ...
->with($this->matchesRegularExpression(
    '/Oh how I love (regex|Regular Expressions)/'
));

So that’s pretty cool, if you happen to like really obscure features that are impossible to remember.

Surely there’s a better way? Think of the children!

What if you could just ask for all the invocations and test that they were right in that language you’re already using for all your production logic? Wouldn’t that be just dandy!

Turns out you can, but it’s hiding – and I don’t mean it’s hiding in a “you will find this if you read the manual” kind of way, I mean it’s hiding in the source code, where everyone totally looks first for easy examples right?

All you have to do is store the result of $this->any() and you can use it as a spy:

1
2

$exec->expects($spy = $this->any())
    ->method('foo');

(I’ve got to wonder if documenting those extra 7 characters might be the colloquial straw that breaks the PHPUnit manual’s back.)

Now that you have a spy, you can just do normal stuff that calls it, then use normal PHP logic (I had to laugh when I wrote “normal PHP logic”) to confirm it’s right:

1
2
3

// get the last invocation
$invocation = end($spy->getInvocations());
$this->assertEquals('foo', $invocation->arguments[0]);

An Example You Say?

As a concrete example, lets ensure the NSA is spying on its citizens just the right amount.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

<?php
// What we're testing today
class AverageCitizen {
    public function spyOn() {}
}

// Our tests (yes, normally these would be in some other file)
class TestAverageCitizens extends PHPUnit_Framework_TestCase {
    public function testSpyingLikeTheNSAShould() {
        $citizen = $this->getMock('AverageCitizen');
        $citizen->expects($spy = $this->any())
            ->method('spyOn');

        $citizen->spyOn("foo");

        $invocations = $spy->getInvocations();

        $this->assertEquals(1, count($invocations));

        // we can easily check specific arguments too
        $last = end($invocations);
        $this->assertEquals("foo", $last->parameters[0]);
    }

    public function testSpyingLikeTheNSADoes() {
        $citizen = $this->getMock('AverageCitizen');
        $citizen->expects($spy = $this->any())
            ->method('spyOn');

        $citizen->spyOn("foo");
        $citizen->spyOn("bar");

        $invocations = $spy->getInvocations();

        $this->assertEquals(1, count($invocations));
    }
}
?>

and when we run the tests we can see that even PHPUnit knows the NSA has crossed the line:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

$ phpunit --debug test.php 
PHPUnit 3.6.10 by Sebastian Bergmann.


Starting test 'TestAverageCitizens::testSpyingLikeTheNSAShould'.
.
Starting test 'TestAverageCitizens::testSpyingLikeTheNSADoes'.
F

Time: 0 seconds, Memory: 3.25Mb

There was 1 failure:

1) TestAverageCitizens::testSpyingLikeTheNSADoes
Failed asserting that 2 matches expected 1.

/i/be/a/coder/test.php:35

FAILURES!
Tests: 2, Assertions: 4, Failures: 1

If you want the % character in a command, as part of a cronjob:
1. You escape the %, so it becomes \%
2. echo and pipe the command you want to run (with the escaped %) into sed
3. Have sed unescape the %
4. Pipe it into the original program

Which I responded to with roughly “if you want a ‘%’ in your cron line you actually want a shell script instead”… this turns out to be a great argumentdebate as a lot of very good Sys Admins (who were online at the time) completely disagreed with me until I’d spelled out my argument in more detail.

Problems with cron

• The syntax can be quite insane if you’re expecting it to behave like shell (hint: cron != shell)
• There’s no widely used crontab linter (I was going to leave it at “there’s no crontab linter”, but found chkcrontab while writing this, which looks like a good start but isn’t packaged for any distro I’ve checked yet)
• Badly breaking syntax in a crontab file will cause all jobs in that file to stop running (usually with no error recorded anywhere)
• Unless you’re double entering your scheduling information you’re not going to be able to pick up the absense of the job in your monitoring solution when it fails to run
• I’m stupid (yes this is a problem with cron)

All of these have led me to break cron lots of times and even more times I’ve had to try to figure out why a scheduled job isn’t running after someone else has broken it for me. Happy days.

KISS

Whenever I’m breaking something fairly critical too often for comfort, it’s time to Keep It Simple Stupid and the way I’ve tried to do that with cron is to never ever put anything complicated on a cron line.

Lets take a simple example:

1

* * * * * echo % some % percents % for % you %

Intuitively I’d just expect that to do what it does on the shell (echo’s back the string, which in cron would normally make it back to someone in email form), but instead the first % will start STDIN for the command, the remaining %s will get changed to new lines and you’ll end up with an echo statement that just echo’s a single new line out to cron as it’s not interested in the STDIN fed to it.

This creates a testing problem because now to test the behaviour of the cron line I need to wait for cron to run the cron line (there’s no way to immediately confirm the validity of the line).

If we instead place the behaviour we want in a script:

1
2

#!/bin/bash -e
echo % some % percents % for % you %

and call that from cron:

1

* * * * * /path/to/script

You can be reasonably confident that it’ll do exactly the same thing when cron runs it as when you test it on the terminal.

But % is ok when it’s simple

Some people tried to make the argument that a % is really ok when it’s actually really simple, e.g.:

1

* * * * * date +\%Y\%m\%d_\%H\%M\%S > /tmp/test

happens to work the same if you copy it to a terminal because the %s are escaped in the cron line and the escaping will happen to drop off in shell as well, but what if you want a quoted % – you’re stuffed.

Back to KISS again.

Other reasons to keep cron simple

If you’re editing cron via crontab -e it’s far too easy to wipe out your crontab file.

While this is mostly an argument for backups, if you keep your cron files simple it may not matter as much when they get nuked accidentally as now you’ve only lost scheduling information and not critical syntax :)

Summary

If I’m not 100% certain I can copy a line out of cron and run it on the terminal I think it doesn’t belong in cron.

Rather than just sit around and complain or try to get stuff in to the core (where there’s no way I’d be able to use it in real world projects until RHEL catches up, i.e. 3-4 years from now) I thought I’d see what I could do purely in PHP.

Turns out it’s quite a lot, so it’s up on Github: https://github.com/neerolyte/php-lyte-xml#readme

So if you have to deal with the XML in PHP fairly often consider taking it for a spin.

This is a great idea, but it’s sorely missing one core feature for anyone who works on more than one machine – the ability synchronise the stashes between machines, so if you’re like me (I work on the same code on up to about 4 individual machines in a week) you probably want some way to move stashes around.

So I’ve started git-rstash, as usual it’s written in terrible bash in the hope that someone will take enough offence at it to take the whole problem off my hands, in the mean time maybe you’ll find it useful too.

For the moment synchronising them is purely up to the user, but they are conveniently placed where the user can drop them in whatever cloud-syncy-like thing they’re already using (Unison, Ubuntu One, Dropbox, etc).

1
2
3
4
5
6

base64_decode() {
  php -r 'echo base64_decode(stream_get_contents(STDIN));'
}
base64_encode() {
  php -r 'echo base64_encode(stream_get_contents(STDIN));'
}

Which can be used to encode or decode base64 strings in a stream, e.g.:

1
2
3
4

$ echo foo | base64_encode
Zm9vCg==
$ echo Zm9vCg== | base64_decode
foo

which is fun, but I wanted to make it a little more portable, so lets try a few more languages…

1
2
3
4
5
6
7
8
9
10
11
12

# ruby
base64_encode() { ruby -e 'require 'base64'; puts Base64.encode64(ARGF.read)'; }
base64_decode() { ruby -e 'require 'base64'; puts Base64.decode64(ARGF.read)'; }
# python
base64_encode() { python -c 'import base64, sys; sys.stdout.write(base64.b64encode(sys.stdin.read()))'; }
base64_decode() { python -c 'import base64, sys; sys.stdout.write(base64.b64decode(sys.stdin.read()))'; }
# perl
base64_encode() { perl -e 'use MIME::Base64; print encode_base64(<STDIN>);'; }
base64_decode() { perl -e 'use MIME::Base64; print decode_base64(<STDIN>);'; }
# openssl
base64_encode() { openssl enc -base64; }
base64_decode() { openssl enc -d -base64; }

and now to wrap them all under something that picks whichever seems to be available:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

base64_php() { php -r 'echo base64_$1(stream_get_contents(STDIN));'; }
base64_ruby() { ruby -e 'require 'base64'; puts Base64.${1}64(ARGF.read)'; }
base64_perl() { perl -e 'use MIME::Base64; print $1_base64(<STDIN>);'; }
base64_python() { python -c 'import base64, sys; sys.stdout.write(base64.b64$1(sys.stdin.read()))'; }
base64_openssl() { openssl enc $([[ $1 == encode ]] || echo -d) -base64; }
base64_choose() {
  for lang in openssl perl python ruby php; do 
    if [[ $(type -t '$lang') == 'file' ]]; then
      'base64_$lang' '$1'
      return
    fi
  done
  echo 'ERROR: No suitable language found'
  return 1
}
base64_encode() { base64_choose encode; }
base64_decode() { base64_choose decode; }

great, now I can quickly grab some base64 commands on any box I’m likely to be working on in the foreseeable future.

Finding an answer that worked around it for singular VMs, I wanted something that worked for all VMs on the laptop (at least until I can fix the actually broken DNS resolver) and I’ve found one.

As per http://docs-v1.vagrantup.com/v1/docs/vagrantfile.html, it’s possible to specify additional parameters to Vagrant in ~/.vagrant.d/Vagrantfile – but it’s not exactly clear how, turns out you can just place them in a normal config block like so:

1
2
3

Vagrant::Config.run do |config|
    config.vm.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end

Before applying the fix I was getting hangs at “Waiting for VM to boot” like so:

1
2
3
4
5
6
7
8
9
10

$ vagrant up
[default] VM already created. Booting if it's not already running...
[default] Clearing any previously set forwarded ports...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.

after applying the fix, it continues on:

1
2
3
4
5
6
7

...
[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
[default] Configuring and enabling network interfaces...
[default] Setting host name...
[default] Mounting shared folders...
[default] -- v-root: /vagrant

Edit: I just thought I’d add that the config has changed slightly with Vagrant 1.1+, so you now need a “v2 config block” (see: http://docs.vagrantup.com/v2/virtualbox/configuration.html):

1
2
3
4
5

Vagrant::configure('2') do |config|
  config.vm.provider "virtualbox" do |v|
    v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
  end
end

If for whatever reason a do-release-upgrade of Ubuntu fails part way through (say because you’ve used 75% of metadata and btrfs doesn’t seem to fail gracefully) and you’ve got a working apt-snapshot, after you get everything back in order again it will have left behind a snapshot that you no longer want:

1
2
3
4

root@foo:/# btrfs subvol list /
ID 256 top level 5 path @
ID 257 top level 5 path @home
ID 259 top level 5 path @apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28

So you think to yourself, “ok I can just delete it now”:

1
2
3
4

root@foo:/# btrfs subvol delete @apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28
ERROR: error accessing '@apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28'
root@foo:/# btrfs subvol delete /@apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28
ERROR: error accessing '/@apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28'

… hmm the obvious things don’t work.

Turns out when a subvolume (in this case “@”) is mounted the snapshot subvolumes aren’t mounted anywhere and you actually have to give a place where it’s visible as a subvolume (not a direct mount).

Because that sentence made no sense, here’s an example, this doesn’t work:

1
2
3
4

root@foo:~# mkdir /@apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28
root@foo:~# mount -t btrfs -o subvol=@apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28 /dev/mapper/foo-root /@apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28
root@foo:~# btrfs subvol delete @apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28
ERROR: error accessing '@apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28'

because I’ve mounted the subvolume I want to delete and I’m giving the top of a FS to the subvolume delete command.

Instead, here’s what does work (even with the FS already mounted on /), create somewhere to mount it:

1

root@foo:/# mkdir /mnt/tmp

Mount it:

1

root@foo:/# mount /dev/mapper/foo-root /mnt/tmp

Show that the subvolumes are all available as directories under the mount point:

1
2
3
4
5

root@foo:/# ls -l /mnt/tmp/
total 0
drwxr-xr-x 1 root root 292 Mar  8 10:26 @
drwxr-xr-x 1 root root 240 Feb 21 08:31 @apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28
drwxr-xr-x 1 root root  14 Mar  8 10:24 @home

Delete it:

1
2

root@foo:/# btrfs subvol delete /mnt/tmp/@apt-snapshot-release-upgrade-quantal-2013-03-07_21\:34\:28
Delete subvolume '/mnt/tmp/@apt-snapshot-release-upgrade-quantal-2013-03-07_21:34:28'

Hooray! It didn’t error this time, it also actually worked:

1
2
3

root@foo:/# btrfs subvol list /
ID 256 top level 5 path @
ID 257 top level 5 path @home

Clean up:

1

root@foo:/# umount /mnt/tmp

… must be something to do with having to fire up a Windows VM to talk to a Linux based phone.

I decided to talk about Vagrant and VeeWee. I don’t claim to be any kind of expert, but I have been tinkering and thought maybe I could stir a little more local interest in these types of automation tools.

Audio

Talk audio is available (50MB OGG).

For the most part the videos below were what was demoed during the talk, but there are some questions discussed during the talk that may be helpful (and some that simply will not make sense if you weren’t there :p).

Videos

First up I go through  a standard Vagrant Init process, Vagrant is already installed but otherwise it just demos the minimum number of steps to get a VM up and running, then running something dubious from the internets and returning the system to a clean state.

apt-cacher-ng demo – shows what should now be the standard steps for trialling something with Vagrant (git clone, vagrant up). In this case apt-cacher-ng, which can be used to speed up system building by caching repositories of various types when you are rebuilding similar machines quite often (https://github.com/neerolyte/mlug-vm-demos):

Spree Install – a nice complex rails app that grabs things off all sorts of places on the internet, sped up slight by using the apt-cacher-ng proxy above, but of course with Vagrant we still just “vagrant up” (https://github.com/neerolyte/mlug-vm-demos):

VeeWee build of a Vagrant basebox with a wrapper script (https://github.com/neerolyte/lyte-vagrant-boxes):

Resources

• Vagrant: http://vagrantup.com/
• VeeWee: https://github.com/jedi4ever/veewee
• Vagrant Boxes: http://www.vagrantbox.es/
• Mozilla blog article: http://blog.mozilla.org/webdev/2011/10/04/developing-with-vagrant-puppet-and-playdoh/
• My VeeWee basebox builder: https://github.com/neerolyte/lyte-vagrant-boxes
• My demo Vagrant specs: https://github.com/neerolyte/mlug-vm-demos
• My Vagueant project (vaguely Vagrant for LXC): https://github.com/neerolyte/vagueant

The reasons I had (broadly) for starting repoproxy:

- shared caches - multiple caches on a LAN, they should share some how

- CentOS - CentOS uses yum, how could it possibly be supported by ACNG?

- roaming clients - e.g. laptops that can't always sit behind the same static instance of ACNG

- NodeJS - I wanted to learn NodeJS, I've learnt a bit now and don't feel a huge desire to complete this project given I've figured out most of the other features neatly

Below I cover how to achieve all of the above features with ACNG as it really wasn’t obvious to me so I suspect it might be useful to other too.

Shared Caches

I run a lot of VMs on my laptop, but also have other clients in the home and work networks that could benefit from having a shared repository cache. For me a good balance is having static ACNG servers at both home and work, but also having ACNG deployed to my laptop so that the the VMs can be updated, reimaged or get new packages without chewing through my mobile bandwidth.

This is actually natively supported with ACNG, it’s just a matter of putting a proxy line in /etc/apt-cacher-ng/acng.conf like so:

1

Proxy: http://mirror.lyte:3142/

Then it’s just a matter of telling apt to use a localhost proxy anywhere that ACNG is installed:

1

echo 'Acquire::http { Proxy 'http://localhost:3142/'; };' > /etc/apt/apt.conf.d/01proxy

This allows VMs on my laptop to have a portable repository cache when I’m not on a normal network, but also allows me to benefit from cache others generate and vice versa.

I’d like to at some point have trusted roaming clients (i.e. only my laptop) publish captured ACNG cache back to static ACNG cache servers. I’m pretty sure I can achieve this using some if-up.d trickery combined with rsync, but I haven’t tried yet.

I had considered trying to add something more generic to repoproxy that did a cache discovery and then possibly ICP (Internet Cache Protocol) to share cache objects between nodes on the same LAN, but there are some generalised security issues I can’t come up with a good solution for, e.g. If my laptop is connected to a VPN and another node discovers it, how do I sensibly ensure they can’t get anything via the VPN without making the configuration overly obtuse?

It seems like trying to implement self discovery would either involve a lot of excess configuration on participating nodes, or leave gaping security holes so for the moment I’m keeping it simple.

CentOS

I use CentOS and Scientific Linux sometimes and I’d like their repos to be cached too.

I had originally falsely assumed this would simply be impossible, but I read somewhere that there was at least a little support.

In my testing some things didn’t work out of the box, but could be worked around.

Essentially it seems like ACNG treats most files as one of volatile, persistent or force-cached and it’s just a matter of relying on tweaking URL based regexs to understand any repositories you want to work with.

Roaming Clients

When I move my laptop between home and work or on to a public wifi point I want apt to “just work” I don’t want to have to remember to alter some config each time I need it.

I found two methods described on help.ubuntu.com that were sort of on the right track, but running a cron every minute that insights network traffic when my network is mostly stable seems like a bad idea, as does having to reboot to gain the new settings (especially as Network Manager won’t bring wireless up until after I login, so it wouldn’t even work for my use case).

NB: I’ve already gone back and added my event driven method to the help.ubuntu.com article.

I suspect it would be possible to utilise ACNG’s hook functionality:

8.9 How to execute commands before and after going online? It is possible to configure custom commands which are executed before the internet connection attempt and after a certain period after closing the connection. The commands are bound to a remapping configuration and the config file is named after the name of that remapping config, like debrep.hooks for Remap-debrep. See section 4.3.2, conf/.hooks and /usr/share/doc/apt-cacher-ng/examples/.hooks files for details.

I couldn’t immediately bend this to my will, so I decided to go down a route I already understood.

I decided to use if-up.d to reset ACNG’s config every time there was a new interface brought online, this allows for an event-driven update of the upstream proxy rather than relying on polling intermittently or a reboot of the laptop.

Create a new file /etc/network/if-up.d/apt-cacher-ng-reset-proxy and put the following script in it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

#!/bin/bash

# list of hosts that the proxy might be running on
hosts=(
        acng.on.my.home.network
        acng.on.my.work.network
        acng.at.my.friends.place
)

set_host() {
        host=$1
        line='Proxy: http://$host:3142/'
        if [[ -z $host ]]; then
                line='# Proxy: disabled because none are contactable'
        fi

        # adjust ACNG configuration to use supplied proxy
        sed -i -r 's%^\s*(#|)\s*Proxy: .*$%$line%g' \
                /etc/apt-cacher-ng/acng.conf

        # if apt-cacher-ng is running
        if service apt-cacher-ng status > /dev/null 2>&1; then
                # restart it to take hold of new config
                service apt-cacher-ng restart
        fi
        exit 0
}

try_host() {
        host=$1
        # if we can get to the supplied host
        if ping -c 1 '$host' > /dev/null 2>&1; then
                # tell ACNG to use it
                set_host '$host'
        fi
}

# Run through all possible ACNG hosts trying them one at a time
for host in '${hosts[@]}'; do
        try_host '$host'
done

# no proxies found, unset upstream proxy (i.e. we connect straight to the internet)
set_host

Make sure to adjust the script for your environment and make it executable with:

1

chmod +x /etc/network/if-up.d/apt-cacher-ng-reset-proxy

So uhm, before doing anything dangerous to a drive you’ve taken a backup right?

Moving on…

I chose to use a process that let me do a live migration underneath the file system while there was still data being written to the file system… you know, for fun.

Adding in the new device is as simple as:

1

$ btrfs device add $new_device $existing_mount_point

Which is quick! Suddenly you have a whole bunch more data available.

Removing the old device takes a lot longer but the command is just as simple:

1

$ btrfs device delete $old_device $existing_mount_point

To get a vague idea of progress watch:

1
2
3
4
5
6
7

$ btrfs-show
Label: none  uuid: 05d0397c-2dfc-4c2b-a508-22bf23099776
        Total devices 2 FS bytes used 682.24GB
        devid    1 size 931.48GB used 688.29GB path /dev/sdb1
        devid    2 size 1.82TB used 13.00GB path /dev/sdd

Btrfs Btrfs v0.19

In this case /dev/sdb1 is the old drive and /dev/sdd is the new drive. The usage of the new drive will gradually grow until it catches up with the old one. In the capture above it’s about 19% (13.00GB / 688.29GB) complete.

Eventually when it completes /dev/sdb1 no longer shows in the device list:

1
2
3
4
5
6

$ btrfs-show
Label: none  uuid: 05d0397c-2dfc-4c2b-a508-22bf23099776
        Total devices 1 FS bytes used 682.04GB
        devid    2 size 1.82TB used 686.52GB path /dev/sdd

Btrfs Btrfs v0.19

It’s dangerous – you’re placing a socket that will happily answer cryptographic challenges that it shouldn’t be in the hands of unknown parties.

It’s only marginally easier than the alternative.

So what is it?

Well as usual the man page is fairly helpful here:

1
2
3
4
5
6
7
8
9

 -A      Enables forwarding of the authentication agent connection.  This can also
         be specified on a per-host basis in a configuration file.

         Agent forwarding should be enabled with caution.  Users with the ability
         to bypass file permissions on the remote host (for the agent's
         UNIX-domain socket) can access the local agent through the forwarded con‐
         nection.  An attacker cannot obtain key material from the agent, however
         they can perform operations on the keys that enable them to authenticate
         using the identities loaded into the agent.

The normal reason why you would want to forward your agent is because you’re connecting to some server via some other server (usually because a direct connection isn’t available).

The diagram below shows a fairly standard scenario where a DB server can’t be accessed via the public internet, but can be indirectly accessed via the web server:

What a lot of people do at this point is forward their key with something like:

1
2
3

yourlaptop$ ssh -A you@webserver
webserver$ ssh you@dbserver
dbserver$

Which is relatively simple (and can be simplified more so by turning on authentication agent forwarding in ssh_config), but it’s terrible unless you completely trust the security of your web server (please don’t do that).

Lets look at why it’s so terrible:

1. Anyone else with sufficient access to the web server can use your forwarded agent to authenticate to anything you have access to.
2. Most web servers have known vulnerabilities that you probably haven’t patched for yet.
3. 1 + 2 = terrible

To be clear I’m not saying that we’re forwarding actual bits of key material to an untrustworthy host, just that we’re forwarding a connection to an agent that is able to answer authentication challenges to another host you may have access to. In practice one way attackers might use this is if they can see the environment variables your SSH connection came in via (either they have gained access to your account or root) then they could interrogate the SSH_AUTH_SOCK environment variable, add that in to their own environment and use your authentication agent to initiate authenticated connections to other hosts while you are still connected to the untrustworthy host.

So what’s the alternative?

ProxyCommand + nc.

ProxyCommand is a (seemingly little known) directive you can give to SSH to say “hey before you connect to where I’ve told you to, fire up a socket to something arbitrary first”. “nc” (or netcat) is something to give you an arbitrary connection.

Here’s an example (continuing on from above):

1
2

yourlaptop$ ssh -o 'ProxyCommand ssh you@webserver nc %h %p' you@dbserver
dbserver$

What this does is connects to webserver, authenticates using your key as you and then runs “nc” replacing in the hostname and port for the dbserver as it does. This will return a socket (running over the encrypted SSH connection) that can connect directly to the dbserver. SSH will then connect to the socket as you and authenticate with your key, but note that the encryption between you and the dbserver is end-to-end so you don’t need to make your authentication agent available in any potentially untrustworthy environments.

Ok that’s great and all but it’s too much to type

Put it in your ~/.ssh/config file, for the example above something like this should work:

1
2

Host dbserver
ProxyCommand ssh you@webserver nc %h %p

Once that’s in place you can just run:

1
2

yourlaptop$ ssh you$dbserver
dbserver$

What about a range of hosts?

The only real thing to watch out for here is that if you match a range of hosts and the host you’re routing via is in that matched range, SSH will quite happily fork bomb your machine:

1
2
3
4
5

# Be very careful to disable ProxyCommand on the host you are routing via or you will fork bomb yourself.
Host gateway.difficult.to.get.to
ProxyCommand none
Host *.difficult.to.get.to
ProxyCommand ssh you@gateway.difficult.to.get.to nc %h %p

What if I have to jump via multiple hosts?

Well firstly, redesign your network.

If for some reason that’s not a realistic option, you can just stack hops together:

1
2
3
4
5
6
7

Host hop2
ProxyCommand ssh you@hop1 nc %h %p
Host hop3
ProxyCommand ssh you@hop2 nc %h %p
Host hop4
ProxyCommand ssh you@hop3 nc %h %p
...

But it’s so slow…

In some circumstances doing this can be slower (e.g. when you spend all your time connected to an intermediate server connecting to other servers). This is because you’re now firing up multiple SSH connections each time you make a new connection instead of just one.

One pretty common work around for this is:

1
2
3

ControlMaster auto
ControlPath /home/you/.ssh/master-%r@%h:%p
ControlPersist yes

Run “man ssh_config” and search for “ControlMaster” to read up in more detail about this.

The TL;DR of it is – ControlMaster creates multiplexing magic to convert multiple connections down to one (meaning much less build-up / tear-down lag).

Similarity to onion routing

Using ssh like this is very similar to onion routing in that you have end-to-end encryption (and authentication!) by only trusting each mid-point to run up a socket (courtesy of netcat) for you. However please don’t assume that because the encryption going on here is similar to TOR that you can use it to alleviate the privacy concerns that TOR does.

Update: clarified that the agent and not the key is being forwarded, also explained a little more depth what that means.

It took me to see a waterfall

in Toora

I parked it against a tree

I spotted windmills

and it took me there too

I parked it against a dumpster

I went to see a friend

I parked it against a pole

I went home and parked it against that too

I don’t understand why Aprilia would make such a great bike and then almost ruin it by putting far too much of it on the wrong side of the stand… but they did

fortunately I made mine a hybrid and now it stands on its own

Yeh, that’s my story.

P.s. for those who actually want technical information…

I swapped out the stock stand for one off a Honda CBR 1100 Blackbird. If you attempt this be prepared to completely and utterly disable the side stand kill switch.

Mainly I worked on the Stripe CTF (now closed) and I’m playing with Smash the Stack’s IO.

I’ve already learnt a few things along the way that I thought would have been useful to me in a cheat sheet like form, so here it is…

GCC

When compiling little snippets of C locally on my laptop I’ve found that quite often I’ve had to turn off a fair bit of protection to get the snippets to compile in just the right way.

To get 32 bit binaries (CTFs so far have been 32bit): $ gcc -m32 ... Note: you may need to install gcc-multilib on Ubuntu.

To turn off stack protection (are these canary values?): $ gcc -fno-stack-protector ...

To enable debugging symbols for use with GDB: $ gcc -ggdb ...

Normally GCC will produce binaries with stacks that are not executable, to enable executable stacks: $ execstack -s %binary_created_with_gcc%

Shellcode

Once you’ve figured out what shellcode is and why you need it, you’ll still need some way to store it in a string (probably a program argument), here’s the two main methods I’ve been using.

Using perl (seems to be a favourite amongst the community): $ /some/binary "$(perl -e 'printf "\x90" x 100')"

Straight from bash: $ /some/binary $'\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90'

objdump Figuring out a binaries format: $ objdump -t /foo/bar | grep 'file format' /foo/bar: file format elf32-i386

Getting the address of “main” from the symbol table: $ objdump -t /foo/bar | grep main 080483b4 g F .text 00000067 main

It’s the first properly security related thing I’ve ever been to, but I was amazed just how quickly I was able to catch on to many of the modern techniques being used to pen test systems out in the wild right now. I don’t think it’s because I’m super smart, I think it’s because most of the flaws that are exploitable are super obvious and you really only need one piece of broken logic to get in to somewhere you don’t belong. It was also quite impressive just how creatively a lot of the people there were able to look at 2 or 3 relatively minor bugs and escalate them in to a full system access hack.

Pen testing also isn’t all the low level buffer over flow stuff I was expecting either, sure there are guys there that can do that and other much fancier stuff to get out of buffer overflow, smash your stack and completely own your box, but there’s still the good ol’ “hmm that acl doesn’t look quite right”.

I managed to catch both talks by Adam and James from Insomnia, what was impressed on me most was that I understood almost everything these guys were doing. Further more a lot of what James was talking about to escalate privileges in a Windows environment was stuff that I was tinkering (sometimes a little too successfully with) back at high school :/

Another pretty interesting stream is just how prolific SQL injection is. I thought (before Thursday) that parameterised SQL queries had solved all the injection problems and the world of IT Security had moved on to harder targets. Turns out I was wrong. This seems to be mainly because half of programmers out there still haven’t actually heard about SQL injection :( It’s compounded by the fact that those that have probably don’t realise just how easy tools like sqlmap make it. With sqlmap you can take a blind SQLi (SQL injection) attack and have it fire up a SQL prompt for you that lets you write arbitrary select queries, or dump the DB to CSV… wow. Better yet, even parameterised SQL normally can’t parameterise the fields in the “ORDER BY” so a lot of programmers will be utilising parameterised SQL thinking it makes them completely safe, when in fact they’ve just made it fractionally harder.

There was way too much other cool stuff to talk about, I’m definitely going to try and go again next year.

A fair while ago (at least 2 years) I started using Subversion to do this. At some point I wrote a script that can be put in my user’s cron file to automate updates and commits.

I figured I would finally publish it so that others can use or improve (or simply abuse) it.

I also decided to finally set up a github account, so it can be found there there.

To use it I put lines like this in “crontab -e”:

1
2

# pidgin - 1 day of lag ok
49 */2 * * * . /home/foo/.ssh/agent; /home/foo/path/to/subversion_auto_sync --max-lag=$((24*60*60)) /home/foo/.purple

Which every 2 hours runs subversion_auto_sync on my home directory with my ssh-agent environment loaded to allow automated (but secure) key based authentication. This example will not run an update for up to 24 hours since the last update, but it will always commit if local modifications have occurred. This provides gradual synchronisation of my Pidgin logs between all Desktops/Laptops I use.

I’ve got the vast bulk of my important home directory files either in Subversion using something similar to what’s above, or being synchronised with Unison.

I originally had something like:

1
2
3
4

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          26      208813+  83  Linux
/dev/sda2              27        3943    31463302+  8e  Linux LVM
/dev/sda3            3944        5221    10265535    7  HPFS/NTFS

Note: partition table above is from a VM I spun up to write this entry.

That is to say I had a drive dual booting Linux and Windows, with Windows at the end of drive (because Russel sort of implied it was better).

The person who owns the drive in question wanted all the space claimed by Linux reallocated back to Windows.

After confirming backups of critical documents I fired up gparted in a live CD… which all seems swimmingly ok until the Laptop powered off (because the power plug had dropped out and the battery went flat).

After trying to boot up I discovered a corrupted partition table.

This would probably be ok (given the backups of documents) if not for the fact that I had no appropriate Windows installation media and the owner of the Laptop couldn’t find the original installation disks.

Oh noes! (I scream loudly inside my head so as not to alarm the owner of the Laptop)

After booting back in to the live CD I have a partition table that looks more like this:

1
2
3

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          26      208813+  83  Linux
/dev/sda3              27        1304    10265535    7  HPFS/NTFS

“Ok” I hear you saying “just rewrite the partition table with the output you saved from “fdisk -l” before starting. “Oh you didn’t save the output before starting…”

After a bit of hair pulling (and googling) I come across gpart.

It found the old partition without breaking a sweat:

1
2
3
4
5
6
7
8
9
10
11
12
13

root@ubuntu:~# gpart /dev/sda

Begin scan...
Possible partition(Linux ext2), size(203mb), offset(0mb)
Possible partition(Windows NT/W2K FS), size(10024mb), offset(203mb)
Possible partition(Windows NT/W2K FS), size(10024mb), offset(30929mb)
End scan.
[...]
Primary partition(3)
   type: 007(0x07)(OS/2 HPFS, NTFS, QNX or Advanced UNIX)
   size: 10024mb #s(20531070) s(63344295-83875364)
   chs:  (1023/254/63)-(1023/254/63)d (3943/0/1)-(5220/254/63)r
[...]

After talking to fdisk nicely I was able to convince it to create the partition table that gpart was suggesting and subsequently able to mount, resize properly and validate the partition with Windows chkdsk tool… phew, crisis averted.

Next time I’m dding the disk to a USB drive first, no matter how long it takes.