I live off grid with my partner and 2 young kids. We’ve wanted an EV for a while now, but it’s a little challenging to manage off grid as we can’t just charge overnight the way most people who are ready to transition to EVs can. We’re also a fair way from any fast charging infrastructure. I think we’ve made it work, so thought I’d write up some of the thought process and learnings thus far.

Why the Model 3?

To meet range requirements during winter, when charging will be difficult for 2-4 weeks, we wanted a car with enough range to get through a normal week (roughly 350km) on a single charge. Public charging infrastructure is available but 50kW chargers start at 25km / 25 minutes away and faster ones 70km / 1 hour away, so we don’t want to have to burn a lot of time and range doing it multiple times a week.

The only two “affordable” cars we found with adequate range were:

• Tesla Model 3
• Hyundai Kona MY 2021

Tesla have a car maintenance schedule that is within my abilities to manage at home v.s Hyundai who currently only service the electric Kona in Melbourne (a 3-4 hour round trip) and require regular servicing.

The Kona uses LiPo batteries which degrade faster when charged above 80% and the Model 3s manufactured in China use LFP batteries which don’t have the same limitation. Tesla state that the LFP battery “should be charged to 100% whenever convenient in order to maintain long-term battery health” - which I think is mostly about cell balancing and SoC calibration, not directly cell health.

As we’re planning to opportunistically charge from solar, having to manage the 80% SoC limitation of LiPo cells would just be an extra complication and as such the choice was simple - the Model 3 or wait. We didn’t want to wait.

Charging an EV in general

In Australia we’ve settled on the CCS2 connectors:

For charging at home we are interested in the AC Type 2 (aka Mennekes, aka IEC 62196) portion of that connector:

Start with a standard power plug, upgrade the amperage, add in optional 3 phase and a few data pins and this is what you get.

An Electric Vehicle Supply Equipment (EVSE - “charger”) is required to take whatever AC power is available at the site, mix in some signaling data so the car knows how many amps are available and do some safety checks before engaging a contactor (big clicky relay).

Charging an EV off grid

EV batteries are large. In our Model 3 Standard Range Plus, the battery is 50-60kWh (Tesla actually don’t publish the exact specs and I still haven’t verified if we have the unicorn model). Our house battery is 24kWh LFP with 19.2kWh usable. Charging an EV from a house battery is not practical with current batteries.

Off grid solar generally has to be larger than is required on grid for a similar energy usage as it needs to produce enough to supply the house on marginal production days. That means when it’s sunny, there’s a lot of otherwise unused energy available.

If the driving pattern suits (ours does) - EVs and off grid seems like a perfect match.

Unfortunately there’s no off the shelf EVSEs that work with off grid / hybrid inverters to track solar generation - at least that I could find.

Our off grid power system

The brains of our system is a Selectronic SP Pro SPMC482-AU. It turns DC from the battery in to AC for the house and recharges the DC batteries from coupled AC like our Fronius (on grid style) solar inverter or AC inputs like our generator. It keeps track of battery State of Charge (SoC), temperatures and has enough inputs in general to be the thing that’ll alarm first if anything ever does go wrong.

If I thought there was better hardware available than a SP Pro, I would probably have got that instead, but I don’t.

Getting data out of the SP Pro

Unfortunately Selectronic’s software engineering clout doesn’t live up to that of their hardware engineers.

The primary way of monitoring a SP Pro is with Select.live - a poorly managed service that depends on a flakey serial device plugged in to the inverter. I’ve had the Select.live device crash a few times a week on average until a few months ago when they finally seemed to get their update process in order. I also found (reported and confirmed they fixed) a security issue in the web interface that would allow any account holder to access data from any other account holder their installer still had access to. All of which to say, I really don’t trust it.

I’ve already got some better (improved resolution and exposing hidden metrics) monitoring from the Select.live device in the form of https://github.com/neerolyte/select-live-influx.

Given the stability issues of the Select.live I’ve been meaning to replace it with my own device that talks serial to the SP Pro - https://github.com/neerolyte/selpi. I may still get back to this project, but as I was also using it as an excuse to learn Python it was a little slow going and it’s a low priority while the Select.live is stable.

The data I need from the SP Pro is just:

• battery Watts - how much power is currently flowing in or out of the house battery
• battery SoC - how full is the house battery

EVSEs

OpenEVSE

I purchased an OpenEVSE kit from the US and added in the components required to suit an AU 32A power point and a Type 2 for the Model 3.

The basic algorithm I’ve implemented is a loop that does:

1
2
3
4
5
6
7
8
9
10

# calculate spare amps based on how much power is going in to the house battery
spare_amps = 0 - ( house_battery_w / 240 )

# if house battery is fully charged, assume some spare capacity as solar
# generation has probably reduced to match loads
if house_battery_soc > 99.0:
  spare_amps += 5

# Shift EVSE charge rate by how many amps are spare
openevse.set_charge_amps(spare_amps + openevse.get_charge_amps())

There’s a bit more going on in practice e.g constraining the charge amps within what the EVSE can tell the car to do (minimum 6A and maximum 32A) and enabling/disabling the charger, but that’s enough to track solar once active:

The dips are both due to clouds and an intermittent reset from our Fronious inverter. There’s a small delay, but it will track between 6A and 24A in the current config (one component still needs changing out to get 32A).

The OpenEVSE probably can’t meet Australian Standards given the DIY nature, so this is not really a general option for everyone. I’ve also already hit issues with it overheating in the sun causing it to crash. I’ve built a temporary hat for it, but it was overheating on a 25C day and we get some 45C+ days here.

OpenEVSE has its own solar diversion mode that would probably work if I was already using OpenEnergyMonitor. I may still go down this rabbit hole, but as the actual solar tracking is trivial and I already have all the monitoring I want working, it’s a lot of software to learn and I’d still have to write a plugin for the SP Pro or Select.live to work with OpenEnergyMonitor myself. In the mean time tracking is done with https://github.com/neerolyte/openevse-hackery.

Tesla Wall Connector

The Tesla Wall Connector is a fairly capable Type 2 EVSE with the special Tesla button on the plug that opens the charge port. I assumed the button on the plug was pointless convenience as you can just tap the charge port on the car to open it - but because the car latches on to the Type 2 plug, to disconnect I need to click around in the phone app or on the display from the drivers seat and this is more annoying than I’d expected.

I didn’t know if I could control this charger remotely, which is why I went with the OpenEVSE, but after ordering the OpenEVSE, I found https://github.com/mvaneijken/TWCManager-1 - so I’d pick the Tesla Wall Connector in favour of the OpenEVSE for local network control now. This would also allow all power electronics to be AS compliant and installed professionally.

Tesla UMC

The Tesla Universal Mobile Connector (UMC) is an EVSE that comes with the car.

The UMC accepts multiple “tails” that in Australia allows it to supply charge from any single phase supply up to 32A.

The UMC itself has no way to be told to charge at different rates, but Teslas can be controlled from the app and some of the API is well documented.

This opens up the possibility of having a SaaS product that monitors Select.live over the internet and instructs the car directly what rate to charge at with no additional hardware on site. This is also marginally better for power management as the EVSE PWM signal only goes down to 6A, but from the Tesla phone app and in car UI I can drop the rate to 5A.

Zappi

The Zappi is an EVSE designed to track grid tie solar and optimise charging. Unfortunately with no grid connection for its CT to monitor, using the Zappi would require something hacky like configuring the SP Pro to switch at least one of its outputs based on say house battery SoC and monitoring multiple windings of a small load with the Zappi’s CT. A very blunt on/off charging profile that’ll never utilise as much solar as more direct monitoring and would work the house battery much harder than desireable.

I did contact myenergi about potentially using their internal API to control it, but their response was not inspiring:

Technically, yes it can work off grid, but not very well. It limits lots of features and isn’t the way the zappi was designed to operate – I wouldn’t advise it. There’s lots of things you’d have to change, it would be very complicated.

The reason for this is because a lot of our safety features and product functions rely on a constant flow of electricity.

There’s no safety or lack of continuous power issue off grid (my power is dramatically more reliable than near by grid connected houses), there’s just a desire to control the PWM signal on the control pilot pin with a little more finesse than their device can manage on its own.

Alternative approaches

Charging doesn’t have to be fully automated. Simply using a timer or even plugging on days when there seems to be enough sun should work well enough if there’s a sufficiently well configured and maintained auto start generator to catch the days when the weather is surprising. Even just configuring an early warning alarm on the house battery SoC and disconnecting the EVSE manually could be an option. Not many off gridders are going to be away from home while their car is at home charging.

In the future?

I’d like to integrate solar forecasting such as Solcast to better decide whether the EV or house battery should be prioritised. The free API is adequate for personal monitoring and the prediction generally seems accurate enough to be trusted.

Asking technical questions is hard… well, people get it wrong so often that I’m certain it can’t be easy.

There’s a bunch of reasons why questions get asked poorly or not at all, but here’s some common themes I’ve spotted.

P.s yes, I’m guilty of making every mistake here at least once.

XY Problem

How do I get this regex to match the closing HTML tag correctly?

This is probably the most common, it even has its own site.

It’s not uncommon to naively stumble in to a problem that’s way beyond ability and find yourself thinking you can fix it “if only I could solve Y”. The trap here is that if you just ask about Y and don’t put X (the actual problem you’re attempting to solve) in context it can be really hard to for anyone else to figure out you’ve asked the wrong question for you.

Lack of Context

How do I cast to integer?

This is the broader version of the previous problem. Your question may be perfectly reasonable, but without knowing enough about the original problem we have no way to guess what you’re casting or what language you’re in. I mean surely given you’ve used the words “cast” and “integer” you know (int)$foo casts in most languages, so you couldn’t be asking about that… could you?

Asking to ask

Oh I know you’re super busy, I hope you’re having a great day, it’d be great if you could help me with a problem I’m having with my code.

I often just wait to see if these people will ever ask a question.

If you’re going to ask a question, ask it. Asking hasn’t cost the other person anything, answering will and if you make someone work harder to answer you (by asking them to ask you the question you’re trying to ask) they’re going to be less willing to spend that time.

Lack of a Question

Can string a timestamp.

Sometimes someone asks something so garbelled it’s not possible to tell what they actually mean any more.

Sure it lexes and even looks like English, but I have no idea what you were trying to ask. There’s probably a string and a timestamp involved, probably some conversion between the two. Who knows what direction and in what language? Not me.

P.s question marks make it more obvious it was intended as a question.

Afraid to Look Stupid

If anyone thinks you’re stupid for asking a question, they’re wrong. Yes there are many things you could have looked up on Google and some of them might have given you the answer in an entirely reasonable way, but sometimes it’s hard to find the set of search terms that get you that answer.

Caveats: If you ask your question really badly, you can actually come off looking stupid. Don’t use this as an excuse to not try Google first.

Wall-o-text

This is way less common, but you know it if you see it - you just get a wall (pages and pages and pages) of text with no introduction to what the problem is. Lots of info is great, but put some structure around it and introduce the problem first.

Good Questions

It is possible to ask a question well, but it does take effort. Here’s what I think you need to focus on.

Summarise

Start with a short summary of the problem.

You need to set the context, backtrack in your working to what you were actually trying to solve that led you down the path to your current question. Summarise that up front.

Consider the communication medium

A lot of the rest of this advice makes sense for text based chats (email, instant chat, etc). If you’re communicating in person information can flow back and forth much more quickly about what you’ve tried. If you’re communicating over email to someone in another timezone, you likely need to include excruciating detail to not wait a day just to find out you left out something important. If they can’t see your screen, consider including a screen shot, pastebin or codepen as appropriate.

Tell us what you’ve tried

Explain briefly any solutions (if any) you attempted and completely discounted. Explain the solution (if you have one) you are currently attempting and why you think it’s currently not working.

Consider the audience

Has the person you’re asking actually worked on the project you’re currently on or are they a domain expect and completely green to your current problem? This will make a big difference to the background required.

Make it easy

The easier it is to understand your question, the easier it is to answer it. If your question is hard to read, doesn’t include enough info, is clearly the Y part of an XY or any other problem - it’s harder to answer and anything you do to make the job of answering it harder makes it that much less likely someone will bother to answer it.

Have you ever noticed that when git mentions a file it often shows an octal permission mode that doesn’t match with the permission on the file?

No? Try this, from an empty directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Set umask to 0002 (i.e don't mask out user/group bits)
$ umask 0002
# create a new git repo in the current dir
$ git init .
# create a new file
$ touch foo
# get the octal perms on it
$ stat -c "0%a %n" foo
0664 foo
# add and commit
$ git add foo
$ git commit -m "bar"
[master (root-commit) 51dc083] bar
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 foo

So I just added a file to git that has octal perms of 0664 but git reported 100644 - it’s safe to ignore the prefix, but why has the second last digit changed?

Here’s something else to try:

1
2
3
4
5
6
7
8

# Create a file with every (practical) octal mode
$ for file in 0{4..7}{0..7}{0..7}; do touch $file; chmod $file $file; done
# Stage those files
$ git add 0???
# Look at what perms git has staged:
$ git diff --staged | grep '^new file mode ' | sort | uniq -c
    128 new file mode 100644
    128 new file mode 100755

So git took all 256 different combinations of modes we gave it and stored 100644 half the time and 100755 the other half.

This is because git is really only looking at the u+x (user execute aka & 0100) bit - it doesn’t care about the rest of the perms.

I watched a discussion about X-Cache and X-Cache-Lookup headers unfold recently and it turns out a lot of people who I would have thought knew what these headers were indicating were a little muddled up. Further more, it turned out if you go looking for a good explanation, everyone seems to just link to this rather old blog post - despite being well meaning, it’s unfortunately slightly confused too.

So maybe I can give a better explanation.

First question - where is the spec for X-Cache? Well, there isn’t one - that little X- prefix indicates the header is not part of the spec, so its meaning may vary between proxy implementations (in fact with Varnish it’s common to add it yourself in the config file, so it could mean anything at all).

Why am I seeing these headers?

So with that out of the way I’ll focus on where I think you’re most likely to see them, Squid. Squid is a fairly common caching proxy, both as a forward (outbound) caching proxy and as a reverse (inbound/application accelerator/aggregating) proxy. Squid’s doco also fails to clearly define what the headers do, so if you’ve got here because you’re trying to figure out what they mean, there’s a good chance you’ve got a Squid proxy in the mix (even if you didn’t realise it).

What they mean

So, with Squid what these headers mean is:

• X-Cache: Did this proxy serve the result from cache (HIT for yes, MISS for no).
• X-Cache-Lookup: Did the proxy have a cacheable response to the request (HIT for yes, MISS for no).

This means if:

• both are HITs, you made a cacheable request and the proxy had a cacheable response that matched and it handed it back to you.
• X-Cache is a MISS and X-Cache-Lookup is a HIT, you made a request that had a cacheable response but you’ve forced a cache bypass - usually achieved by a hard refresh, commonly activated with Ctrl+F5 or by sending the headers Pragma: no-cache (HTTP/1.0) or Cache-Control: no-cache (HTTP/1.1).
• both are MISSes, you made a request (it doesn’t matter if it was cacheable) and there was no corresponding response object in the proxy cache.

This is completely irrelevant to browser cache - if you’re viewing browser cache and inspecting the headers in Chrome or Firebug then you’ll see what the status of the proxy was at the time the proxy returned it to your browser. Sorry if this is obvious, but a surprising number of people seemed to think that the browser cares and bothers to modify these headers, it doesn’t. Really, it doesn’t. I promise.

How you can test this for yourself

First, if you’re trying to use a browser to inspect headers, understand what it’s really saying, e.g in Google Chrome, look for the (from cache) next to the Status Code: section in the headers:

This means nothing has gone over the network and it brought the object out of browser cache.

All other browsers are left as an exercise for the reader as frankly it’s not the right tool for the job (I just figured I had to cover one, as everyone seems to do it this way anyway).

Fire up a bash terminal and get used to curl.

This is what I usually run:

1
2
3
4

$ curl -sv http://example.com/ 2>&1 > /dev/null | egrep '< (X-Cache|Age)'
< Age: 1
< X-Cache: HIT from example.com
< X-Cache-Lookup: HIT from example.com:80

There’s a lot going on here, so lets break it down:

• curl: this is the main thing we’re running,
• -sv: this enables both --silent and --verbose, which (counterintuitive I know) is the simplest way to get curl in to a mode where it dumps both the body and headers out in a useful format (yes I know you can use --head, but that changes the request format and invalidates enough testing to make it useless).
• http://example.com/: our URL of choice.
• 2>&1: Take STDERR (headers) and redirect it to STDOUT (so we can pipe it to egrep).
• > /dev/null: Ignore the original STDOUT (body).
• | egrep: pipe the headers through egrep.
• '< (X-Cache|Age): grab just headers starting with X-Cache or Age.

You may note I snuck that Age header you’ve been ignoring in there, you can thank me later.

So in this first example we’ve made a cacheable request (absent of Pragma: no-cache or similar headers) and the proxy has had a response from its upstream 1 second ago (Age: 1) that was cacheable.

Now that we have a basic command, lets fiddle with it to see what happens.

If we request the same thing, we should see the Age header count up by roughly the number of seconds since the start of either request:

1
2
3
4

$ curl -sv http://example.com/ 2>&1 > /dev/null | egrep '< (X-Cache|Age)'
< Age: 5
< X-Cache: HIT from example.com
< X-Cache-Lookup: HIT from example.com:80

If we request the same thing but with a hard refresh header (note: Squid and probably every other caching proxy ever can be configured to ignore these headers on the request side, so if you get different results here, 9 times out of 10 that’s why), we can see that the proxy had the object in cache, but didn’t use it:

1
2
3

$ curl -sv --header "Pragma: no-cache" http://example.com/ 2>&1 > /dev/null | egrep '< (X-Cache|Age)'
< X-Cache: MISS from example.com
< X-Cache-Lookup: HIT from example.com:80

If we request something we don’t expect the proxy to have seen before, the proxy neither has it in cache or serves it from cache:

1
2
3

$ curl -sv http://example.com/?"$(date -Ins)" 2>&1 > /dev/null | egrep '< (X-Cache|Age)'
< X-Cache: MISS from example.com
< X-Cache-Lookup: MISS from example.com:80

Sometimes I feel like I should have an entire section on my blog dedicated solely to PHP’s maniacle insanity just so I’d have a place to record whatever crazy new thing I learn about PHP on any day I work with it. Any day I work with it. Ever.

That said, this one particulary annoyed me.

PHP caches stat information. Why? I dunno really, VFS caches stat information too, you’d think it’d do a better job being that people other than PHP core developers designed it (so maybe some of them weren’t drunk at the time), plus it’s actually the right layer, it can see not only file system modifications made by the PHP you’re currently writing, but also all that other PHP you’ve written and the stuff Mr Jones wrote, also that strange bit of code that’s not PHP. So it would actually be able to cache the right things at VFS, instead PHP core devs actually thought it made sense to cache it in a PHP process, but what the hell lets leave OS problems to application developers and just move on.

So back on topic, PHP caches stat information or to put it another way, PHP has a stat cache (do you see what I did there?). Stat information are all those boring statistics you get about files when you write stat <file>, but really it’s so much more. It’s most of the data about a file that isn’t actually the file itself.

Great, so PHP caches stat information. That must make things faster or something? Possibly, but it also just makes it a massive hot squishy steaming pile of lies. Because any interaction with a file’s stat info is cached, if a file is modified in some way outside of PHP that you care about you now have to constantly call clearstatcache() so that when you check anything about the file you can be confident that PHP isn’t lying to you.

Did I mention PHP has a stat cache? A-packed-full-of-lies-but-at-least-it’s-faster-than-telling-you-the-truth-except-now-you-have-to-slow-things-down-by-constantly-clearing-it-stat-cache.

Somewhere about here you learn to deal with it, you think to yourself “it’s ok, I’ll just call clearstatcache() everywhere, it’s fine, it’s just a stupid cache that I can’t turn off, so I may as well deal with it, maybe I’ll write a Vim macro to insert clearstatcache() calls in front of any file system operations, that’ll fix it, it’ll be almost like PHP is a real language”.

Then, because you haven’t written that Vim macro (it’s a stupid idea any way) and you haven’t managed to turn off the broken stat cache (because you can’t) you run in to what must surely be a bug - even PHP core devs couldn’t consider this a feature, surely? You change ownership of a file using the built in chown() function, it’s a glorious thing, at first it’s owned by one user, then it’s owned by another user, cheers go up, standing ovations, it’s possible something built in to PHP actually works as designed. Cool. You go to check this using a PHP function (because like all PHP devs you totally do TDD and as such need to check such things) and because you updated the ownership information using a PHP function obviously the cache that PHP manages should be marked dirty and you’ll get back an accurate result right?

No.

What?

No, you don’t.

I’m going to say it again… What?

Ok it must be a bug.

Oh, no, you see, actually, it’s the first fscking example on the manual page.

Hmmmm….

Did I mention, PHP has a stat cache?

Trying to spy on invocations with PHPUnit seems to normally involve either writing your own spy class:

1
2
3
4
5
6

class IAmASpy {
  public $invocations = array();
  public function foo() {
      $this->invocations []= 'foo';
  }
}

or trying to use execution checks on mock objects to determine that things were called with the right arguments:

1
2
3
4

$mock = $this->GetMock('Foo');
$mock->expects($this->once())
    ->method('bar')
    ->with($this->identicalTo('baz'));

What A Pain!

What if you want to check the arguments going in to the last call? Well you can use at():

1

$mock->expects($this->at(7)) // ...

… better hope we never add any other calls!

What if we don’t know the exact parameter that it’s being called with and want to check it with something more complex? Well if you dig really hard in the manual you’ll find there’s a whole bunch of assertions that let you feed in crazier stuff like:

1
2
3
4

// ...
->with($this->matchesRegularExpression(
    '/Oh how I love (regex|Regular Expressions)/'
));

So that’s pretty cool, if you happen to like really obscure features that are impossible to remember.

Surely there’s a better way? Think of the children!

What if you could just ask for all the invocations and test that they were right in that language you’re already using for all your production logic? Wouldn’t that be just dandy!

Turns out you can, but it’s hiding - and I don’t mean it’s hiding in a “you will find this if you read the manual” kind of way, I mean it’s hiding in the source code, where everyone totally looks first for easy examples right?

All you have to do is store the result of $this->any() and you can use it as a spy:

1
2

$exec->expects($spy = $this->any())
    ->method('foo');

(I’ve got to wonder if documenting those extra 7 characters might be the colloquial straw that breaks the PHPUnit manual’s back.)

Now that you have a spy, you can just do normal stuff that calls it, then use normal PHP logic (I had to laugh when I wrote “normal PHP logic”) to confirm it’s right:

1
2
3

// get the last invocation
$invocation = end($spy->getInvocations());
$this->assertEquals('foo', $invocation->arguments[0]);

An Example You Say?

As a concrete example, lets ensure the NSA is spying on its citizens just the right amount.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

<?php
// What we're testing today
class AverageCitizen {
    public function spyOn() {}
}

// Our tests (yes, normally these would be in some other file)
class TestAverageCitizens extends PHPUnit_Framework_TestCase {
    public function testSpyingLikeTheNSAShould() {
        $citizen = $this->getMock('AverageCitizen');
        $citizen->expects($spy = $this->any())
            ->method('spyOn');

        $citizen->spyOn("foo");

        $invocations = $spy->getInvocations();

        $this->assertEquals(1, count($invocations));

        // we can easily check specific arguments too
        $last = end($invocations);
        $this->assertEquals("foo", $last->parameters[0]);
    }

    public function testSpyingLikeTheNSADoes() {
        $citizen = $this->getMock('AverageCitizen');
        $citizen->expects($spy = $this->any())
            ->method('spyOn');

        $citizen->spyOn("foo");
        $citizen->spyOn("bar");

        $invocations = $spy->getInvocations();

        $this->assertEquals(1, count($invocations));
    }
}
?>

and when we run the tests we can see that even PHPUnit knows the NSA has crossed the line:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

$ phpunit --debug test.php
PHPUnit 3.6.10 by Sebastian Bergmann.


Starting test 'TestAverageCitizens::testSpyingLikeTheNSAShould'.
.
Starting test 'TestAverageCitizens::testSpyingLikeTheNSADoes'.
F

Time: 0 seconds, Memory: 3.25Mb

There was 1 failure:

1) TestAverageCitizens::testSpyingLikeTheNSADoes
Failed asserting that 2 matches expected 1.

/i/be/a/coder/test.php:35

FAILURES!
Tests: 2, Assertions: 4, Failures: 1

Someone brought up from our Sys Admin chat at work yesterday that crontab and % are insane, the summary was:

If you want the % character in a command, as part of a cronjob:
1. You escape the %, so it becomes \%
2. echo and pipe the command you want to run (with the escaped %) into sed
3. Have sed unescape the %
4. Pipe it into the original program

Which I responded to with roughly “if you want a ‘%’ in your cron line you actually want a shell script instead”… this turns out to be a great argumentdebate as a lot of very good Sys Admins (who were online at the time) completely disagreed with me until I’d spelled out my argument in more detail.

Problems with cron

• The syntax can be quite insane if you’re expecting it to behave like shell (hint: cron != shell)
• There’s no widely used crontab linter (I was going to leave it at “there’s no crontab linter”, but found chkcrontab while writing this, which looks like a good start but isn’t packaged for any distro I’ve checked yet)
• Badly breaking syntax in a crontab file will cause all jobs in that file to stop running (usually with no error recorded anywhere)
• Unless you’re double entering your scheduling information you’re not going to be able to pick up the absense of the job in your monitoring solution when it fails to run
• I’m stupid (yes this is a problem with cron)

All of these have led me to break cron lots of times and even more times I’ve had to try to figure out why a scheduled job isn’t running after someone else has broken it for me. Happy days.

KISS

Whenever I’m breaking something fairly critical too often for comfort, it’s time to Keep It Simple Stupid and the way I’ve tried to do that with cron is to never ever put anything complicated on a cron line.

Lets take a simple example:

1

* * * * * echo % some % percents % for % you %

Intuitively I’d just expect that to do what it does on the shell (echo’s back the string, which in cron would normally make it back to someone in email form), but instead the first % will start STDIN for the command, the remaining %s will get changed to new lines and you’ll end up with an echo statement that just echo’s a single new line out to cron as it’s not interested in the STDIN fed to it.

This creates a testing problem because now to test the behaviour of the cron line I need to wait for cron to run the cron line (there’s no way to immediately confirm the validity of the line).

If we instead place the behaviour we want in a script:

1
2

#!/bin/bash -e
echo % some % percents % for % you %

and call that from cron:

1

* * * * * /path/to/script

You can be reasonably confident that it’ll do exactly the same thing when cron runs it as when you test it on the terminal.

But % is ok when it’s simple

Some people tried to make the argument that a % is really ok when it’s actually really simple, e.g.:

1

* * * * * date +\%Y\%m\%d_\%H\%M\%S > /tmp/test

happens to work the same if you copy it to a terminal because the %s are escaped in the cron line and the escaping will happen to drop off in shell as well, but what if you want a quoted % - you’re stuffed.

Back to KISS again.

Other reasons to keep cron simple

If you’re editing cron via crontab -e it’s far too easy to wipe out your crontab file.

While this is mostly an argument for backups, if you keep your cron files simple it may not matter as much when they get nuked accidentally as now you’ve only lost scheduling information and not critical syntax :)

Summary

If I’m not 100% certain I can copy a line out of cron and run it on the terminal I think it doesn’t belong in cron.

XML support in PHP is actually pretty good these days, but as with anything in PHP (why is that?) it has a few little quirks and corner cases that provide for continual facepalm moments.

Rather than just sit around and complain or try to get stuff in to the core (where there’s no way I’d be able to use it in real world projects until RHEL catches up, i.e. 3-4 years from now) I thought I’d see what I could do purely in PHP.

Turns out it’s quite a lot, so it’s up on Github: https://github.com/neerolyte/php-lyte-xml#readme

So if you have to deal with the XML in PHP fairly often consider taking it for a spin.

Git has a feature called stash that lets drop whatever you’re working on and put the working directory back to a clean state without having to commit or lose whatever deltas you had.

This is a great idea, but it’s sorely missing one core feature for anyone who works on more than one machine - the ability synchronise the stashes between machines, so if you’re like me (I work on the same code on up to about 4 individual machines in a week) you probably want some way to move stashes around.

So I’ve started git-rstash, as usual it’s written in terrible bash in the hope that someone will take enough offence at it to take the whole problem off my hands, in the mean time maybe you’ll find it useful too.

For the moment synchronising them is purely up to the user, but they are conveniently placed where the user can drop them in whatever cloud-syncy-like thing they’re already using (Unison, Ubuntu One, Dropbox, etc).

I find myself writing little functions to to paste in to terminals to provide stream handlers quite often, like:

1
2
3
4
5
6

base64_decode() {
  php -r 'echo base64_decode(stream_get_contents(STDIN));'
}
base64_encode() {
  php -r 'echo base64_encode(stream_get_contents(STDIN));'
}

Which can be used to encode or decode base64 strings in a stream, e.g.:

1
2
3
4

$ echo foo | base64_encode
Zm9vCg==
$ echo Zm9vCg== | base64_decode
foo

which is fun, but I wanted to make it a little more portable, so lets try a few more languages…

1
2
3
4
5
6
7
8
9
10
11
12

# ruby
base64_encode() { ruby -e 'require 'base64'; puts Base64.encode64(ARGF.read)'; }
base64_decode() { ruby -e 'require 'base64'; puts Base64.decode64(ARGF.read)'; }
# python
base64_encode() { python -c 'import base64, sys; sys.stdout.write(base64.b64encode(sys.stdin.read()))'; }
base64_decode() { python -c 'import base64, sys; sys.stdout.write(base64.b64decode(sys.stdin.read()))'; }
# perl
base64_encode() { perl -e 'use MIME::Base64; print encode_base64(<STDIN>);'; }
base64_decode() { perl -e 'use MIME::Base64; print decode_base64(<STDIN>);'; }
# openssl
base64_encode() { openssl enc -base64; }
base64_decode() { openssl enc -d -base64; }

and now to wrap them all under something that picks whichever seems to be available:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

base64_php() { php -r 'echo base64_$1(stream_get_contents(STDIN));'; }
base64_ruby() { ruby -e 'require 'base64'; puts Base64.${1}64(ARGF.read)'; }
base64_perl() { perl -e 'use MIME::Base64; print $1_base64(<STDIN>);'; }
base64_python() { python -c 'import base64, sys; sys.stdout.write(base64.b64$1(sys.stdin.read()))'; }
base64_openssl() { openssl enc $([[ $1 == encode ]] || echo -d) -base64; }
base64_choose() {
  for lang in openssl perl python ruby php; do
    if [[ $(type -t '$lang') == 'file' ]]; then
      'base64_$lang' '$1'
      return
    fi
  done
  echo 'ERROR: No suitable language found'
  return 1
}
base64_encode() { base64_choose encode; }
base64_decode() { base64_choose decode; }

great, now I can quickly grab some base64 commands on any box I’m likely to be working on in the foreseeable future.