The Long Tail

It’s a truism to say that the ‘net is ideal for a long tail, of material of interest to someone, but at best very marginal value or interest to the world at large. Vanity publishing, personal pages, personal blogs, social media, etc, etc.

It even has sophisticated Peer Review processes: a page with Google pagerank 9 is one that the world considers a leader in its field, while a PR4 is a long tail and might be of interest if you are looking to solve a very specific problem it happens to address outside the mainstream.  There’s even a competing hierarchy of peer review systems, with middle-ranking systems like the once-useful but perhaps too easy to subvert Tripadvisor and its peers, niche systems designed to measure one’s own effectiveness, and a long tail of wannabes.

But yesterday listening to the radio I was reminded that a long tail of marginal material and even utter dross doesn’t originate online.  If there’s so much that’s great in the Edinburgh fringe, how come the BBC broadcasts so much that’s – to put it very politely – utterly mediocre?

[correction: not yesterday, it was Saturday the Edinburgh crap on Radio 4 really excelled itself]

Fruits of the Garden

I’ve just eaten my first blackberry of the year.

I used to think of blackberries as (predominantly) September fruit. Late August, early October, but predominantly September’s bounty.

In Southwest England I gradually got used to an extended season. In 2003, when I had no money for food, I was picking and eating blackberries into the second half of November. By that time they had long since ceased to be a pleasure, and were hard work to gather, but in the absence of soup kitchens it was a matter of necessity. Subsequently I also found that we have a rich crop through pretty-much the whole of August, too: indeed, on lower ground there’s more in August than September.

But mid-July? This must be a new record! Though it was just the one berry that had reached a real sweet ripeness.

The bushes in my garden show a huge crop to come. Last year I ate them, froze them, gave them away, cooked desserts with them, even made several jars of chutney, and still could’ve had more. This year I have a new fruit&veg juicer, so I look forward to drinking some, too!

And I must check up on the plums. Last year I had just a handful of them from the garden, but they too were delicious.

Stream Editor for Trafficserver

I haven’t blogged much on software of late. Well, I don’t seem to have blogged so much at all, but my techie contents have been woefully sparse even within a meagre whole.

Well, I’ve just added a new stream editor in to Apache Trafficserver.  It’s been on my to-do list for a long time to produce a similar functionality to sed and sed-like modules in Apache HTTPD.  Now I’ve hacked it up, and dropped in in to the main repo at /plugins/experimental/stream-editor/.  I expect it’ll stay in /experimental/ until and unless it gets sufficient real-world usage to prove itself and sufficient demand to be promoted.

The startingpoint for this was to duplicate the functionality of mod_line_edit or mod_substitute, but with the capability (offered by mod_sed but not by the others) to rewrite incoming as well as outgoing data.  Trafficserver gives me that for free, as the same code will filter both input and output.  Some of the more advanced features, such as HTTPD’s environment variables, are not supported.

There were two main problems to deal with.  Firstly, the configuration needs to be designed and implemented from scratch: that’s currently documented in the source code. It’s a bit idiosyncratic (I’ll append it below): suggestions welcome.  Secondly, the trafficserver API lacks a set of utility classes as provided by APR for Apache HTTPD.  To deal with the latter, I hacked it in C++ and used STL containers, in a manner that should hopefully annoy purists in either C (if they exist) or C++ (where they certainly do).

In figuring it out I was able to make some further improvements: in particular, it deals much better than mod_line_edit or mod_substitute with the case where different rules produce conflicting edits, allowing different rules to be assigned different precedences in configuration to resolve conflicts.  And it applies all rules in a single pass, avoiding the overhead of reconstituting the data or parsing ever-more-fragmented buffers – though it does have to splice buffers to avoid the risk of losing matches that span input chunks.  It parses each chunk of data into an ordered (stl) set before actually applying the edits and dispatching the edited data.

/* stream-editor: apply string and/or regexp search-and-replace to
 * HTTP request and response bodies.
 * Load from plugin.config, with one or more filenames as args.
 * These are config files, and all config files are equal.
 * Each line in a config file and conforming to config syntax specifies a
 * rule for rewriting input or output.
 * A line starting with [out] is an output rule.
 * One starting with [in] is an input rule.
 * Any other line is ignored, so blank lines and comments are fine.
 * Each line must have a from: field and a to: field specifying what it
 * rewrites from and to. Other fields are optional. The full list:
 * from:flags:value
 * to:value
 * scope:flags:value
 * prio:value
 * len:value
 * Fields are separated by whitespace. from: and to: fields may contain
 * whitespace if they are quoted. Quoting may use any non-alphanumeric
 * matched-pair delimiter, though the delimiter may not then appear
 * (even escaped) within the value string.
 * Flags are:
 * i - case-independent matching
 * r - regexp match
 * u (applies only to scope) - apply scope match to full URI
 * starting with "http://" (the default is to match the path
 * only, as in for example a <Location> in HTTPD).
 *   A from: value is a string or a regexp, according to flags.
 *   A to: string is a replacement, and may reference regexp memory $1 - $9.
 *   A scope: value is likewise a string or (memory-less) regexp and
 *   determines the scope of URLs over which the rule applies.
 *   A prio: value is a single digit, and determines the priority of the
 *   rule.  That is to say, two or more rules generate overlapping matches,
 *   the priority value will determine which rule prevails.  A lower
 *   priority value prevails over a higher one.
 *   A len: value is an integer, and applies only to a regexp from:
 *   It should be an estimate of the largest match size expected from
 *   the from: pattern.  It is used internally to determine the size of
 *   a continuity buffer, that avoids missing a match that spans more
 *   than one incoming data chunk arriving at the stream-editor filter.
 *   The default is 20.
 *   Performance tips:
 *    - A high len: value on any rule can severely impact on performance,
 *      especially if mixed with short matches that match frequently.
 *    - Specify high-precedence rules (low prio: values) first in your
 *      configuration to avoid reshuffling edits while processing data.
 *  Example: a trivial ruleset to escape text in HTML:
 *   [out] scope::/html-escape/ from::"&" to:"&amp;"
 *   [out] scope::/html-escape/ from::< to:&lt;
 *   [out] scope::/html-escape/ from::> to:&gt;
 *   [out] scope::/html-escape/ from::/"/ to:/&quot;/
 *   Note, the first & has to be quoted, as the two ampersands in the line
 *   would otherwise be mis-parsed as a matching pair of delimiters.
 *   Quoting the &amp;, and the " line with //, are optional (and quoting
 *   is not applicable to the scope: field).
 *   The double-colons delimit flags, of which none are used in this example.


Our next concert is next Sunday (July 5th), when we’re performing Händel’s Israel in Egypt at the Guildhall, Plymouth.

This is a mature, full-length oratorio on a biblical theme. In parts it is similar to the more famous Messiah (and a few numbers are musically very similar between the two works). In other respects it’s different, and one fundamental difference is that this work uses full antiphonal double chorus. We’ll be split across right/left sides of the stage to deliver the effect.

The subject matter is truly biblical. None of the cuddly, merciful God of Constantine (let alone the modern Church of England), but a vindictive warmonger to make the Islamic State look like a holiday camp. This God doesn’t just indulge in holocaust-scale genocide, he glories in it. Much of the music is correspondingly dark, though there are also some gorgeous interludes.

Also of musical/historic interest, this is a very old edition we’re using. In fact the editor was no less than Felix Mendelssohn. Though better-known as a great composer in his own right, Mendelssohn was right in the vanguard of the revival of the Baroque, so this score is living history!

If you like oratorio, you’ll enjoy this concert.

Non-incoming call

These days I have an android ‘phone[1]. Specifically, a Moto G, with android version 4.4.4 according to info. I’m using CSipSimple to enable my home (landline) number on it. Mostly that works fine, and it seems quite rare to have sufficient signal for voice calls but not for SIP. So all’s well, isn’t it?

Yesterday I got an incoming SIP call while I was out. But taking it out of my pocket, the screen wasn’t showing the call, and I had no way to answer it. I typed in my PIN code, and still it was ringing, but still nothing on the screen. Ouch! What’s going wrong?

Having failed to think of anything more sensible, I went straight for the crude approach of power cycling. That has occasionally fixed things when the system appears to overload itself due to too many open apps, or is running warm for no reason I can fathom. Not that that helps with testing incoming SIP calls, so I tried googling, but failed (admittedly without trying very hard) to find reports of similar problems.

Just now I got another incoming SIP call (no caller number, so no answer). This one did display. It was at home, so on wifi. Could that have made all the difference, and if so is there anything I can do to fix the problem when out and about? Or was this some unknown bug that may have been cured by reboot, or pure Heisenbug?

[1] This is not a good thing. I’d much rather have my late lamented Nokia (from the days when Nokia made really good phones) back. But that’s no longer an option: its successor in 2012 the Nokia E6-00 was such a bugridden steaming pile as to be effectively unusable, and the android is a vast improvement on that.

The things they don’t tell you

I used my new toy for the first time yesterday. A fruit&veg juicer. Feed in fruit&veg at the top and collect a thick, rich juice. Lunch with a concoction of carrots, a hunk of cucumber and half a lemon, together with a banana that needed using up. Evening with a more traditional pear and ginger. Various ingredients bought in bigger bags than I’d’ve done in the pre-juicing era.

Before buying it I had done a bit of research online. Would it do a good job? What could I expect to juice? And crucially, would it be so much faff that I’d soon give up using it? The jury is out on the latter: getting rid of the pulp is a bit more of a faff than with an espresso (or percolator) but in a similar ballpark, and general cleaning just means running the fruity parts through the dishwasher.

Overall, it’s certainly less hard work than juicing as I’ve done in the past. But there are some gotchas, like the liquid trickling slowly out. If I take the jug through to the dining table where I’m eating, a little puddle appears in its place as the final juice dribbles through. And it comes out a little warmer than the fruit going in, so best used with ice.

And then there’s the Big One noone mentioned at all in the bumpf or online reviews. That is, literally. It’s a bigger machine than I’d imagined. I now need a bigger fridge to accommodate ingredients for it, and a bigger dishwasher, not to mention a bigger kitchen. It won’t reasonably go in the corner I’d planned alongside the kettle and espresso machine. It can’t go under any wall cupboard or shelf, because while it just-about fits, it needs space above to feed the ingredients in. I’ve finally done some re-arranging so it can live in the far corner between the sink and the wine rack, where it’s also mercifully easy to clean up any little puddles it might make with a simple wipe.

Preliminary verdict: I’m going to enjoy the fruits of this gadget, but it won’t completely replace supermarket juices.

A choice of poisons

How have I failed all this time to post a good rant about the election and its participants?  A plague on all their houses, including the media reporting them and staging silly events.

Well, I have to report, our beloved Prime Minister and his colleagues have accomplished something quite stupendous with their headless chicken act forever aping whatever party is flavour-of-the-month, and their pork-barrelwarehouse blank cheques[1].  They’ve convinced me Miliband is the lesser of two evils!  And that’s despite some of the horrors that surround him (Balls and Harman spring to mind, though they have strong competition from the likes of Pickles, Shapps and Osborne).

Looking to the future, whoever loses will probably have a change of leadership.  Since the worst imaginable outcome is Balls as PM in five years, there’s another reason to consider Miliband now.   Ugh.

As against that, Cameron has one thing going for him.  He’s no leader[2], but his record of holding an uneasy coalition more-or-less together speaks well of his managerial skills.  And his announcement that he won’t serve more than two terms speaks of unusual commonsense.  Blair/Brown, and previously the thoroughly-nasties who undermined Major, might prove mere foreshadowings of how bad things could get within a governing group.

So who can I support?  Well, amongst the parties it’s a clear None of the Above.  They all have some good things to say, but on the overwhelming balance I can only wish a plague on all their houses – including the aspiring minor parties (Green and UKIP) as well as the more established ones.  However, I can look at my local candidates and decide who appears least objectionable.  I’ve done that, and decided my vote is going to a man with a decent background of hard work in a real job, including starting his own company.  But since this is a marginal constituency, and my candidate doesn’t belong to either of the parties with a hope of winning it, he’ll be squeezed and my vote wasted.

[1] Dammit, when the NHS asks for an extra £8billion, that’s supposed to be a bloomin’ negotiating position to start from.  And that’s not even the worst of the wildly reckless pledges: it’s looking increasingly like I’ll reach retirement age with my taxes paying ever more to price me out of housing!  And look at the number of things I didn’t have to mention!
[2] Boris (or Other) might just be.  That remains to be seen.

Der Prozess

Although the title of Franz Kafka’s story of out-of-control nightmare bureaucracy is translated into English as “The Trial”, the original German does it altogether more justice.  A process that turns its practitioners into cogs in a diabolical machine, and plunges victims into helpless limbo.

I had a medical incident last week.  My eyesight vanished very suddenly, and then came back in bits, with times when I could see half a room or a small area of screen or page while the rest was blank nothingness.  It came with a moderate headache, that was unusual in that it took several days to go away.

I’ve suffered a somewhat similar episode before now.  Back in about November 2007 the loss was a little less sudden, but sufficiently similar that I thought I recognised it.  Back then I was alarmed by it and sought urgent medical attention, only to be told I’d have to wait more than two weeks for a GP appointment – the gatekeeper to our medical system.  No amount of protesting urgency would affect that, but they told me to see an optician instead.  I did, and the optician reassured me there was no immediate need for treatment.  So I left it, and indeed my sight returned over a few weeks.

So it was that this time I was not as alarmed as I might have been, and had no expectations of our NHS.

Rather than repeat last time, I posted a “anyone familiar with these symptoms” question to a forum where I had a hope of good answers.  I got some good responses, but the overwhelming message was to get urgent medical attention.  There was even a suggestion of how to bypass the GP.  So with some trepidation I approached our medical establishment.

The contrast with last time couldn’t have been more stark.  NHS 111 told me to get an urgent appointment.  My GP gave me a same-day appointment, and then an urgent referral to the main regional hospital (which is just half an hour by bus from here).

At this point I made my big mistake.  I went straight to the hospital without even returning home to pack for a spell away from home.  So when they kept me in, I found myself without basic personal stuff, and particularly unhappy in pants and a shirt ever-longer past their wash-by date, and not loose enough for long periods of enforced idleness.  My only little luxury was my ‘phone, and earphones that enabled me listen to radio and (largely) block out the many other noises.  Though with that in such heavy use I was faced with an eternal quest to borrow a charger.

Arriving at the hospital with my GP referral, I was received rapidly, and a man came within minutes to run a couple of routine tests (hey, this is great!)  Once he’d come and gone a couple of times I spotted something of a pattern: attention to me was time-sliced with other things, though I know not whether that might involve another patient, paperwork and red tape, or no more than a cuppa and break.  I needed a drink myself, so after checking with him that I had five minutes to spare, I went to one of the shops near the hospital entrance and got something from the chiller.

So far, so good (though the drink wasn’t).  But that was the end of my being attended to.  As five minutes became five hours and more, I made some vain attempts to find out what the **** was going on.  I checked the time of buses home on my ‘phone, and made a particular effort as the penultimate bus time approached, and again an hour later for the last one.  This pattern of waiting in limbo with ever-rising stress levels turned out to be a foretaste of what was to come, and is overwhelmingly the dominant theme of NHS hospital treatment.  I made a semi-successful effort to get comfy enough to doze in a space too small to lie down: dammit, this is like a night on a bench at a station or airport, only with less luxury and space and more noise.

Sometime around 3 a.m. I was desperate for water and to get up and stretch, so I looked around for anyone I could ask about drinking water.  My first attempt failed: he asked what bed I was in and he’d bring me some, and couldn’t understand when I replied that I didn’t have a bed.  So was I visiting someone?  No, I’m waiting to be attended to, and just need some water.  Aaaargh!  Find another member of nursing staff, one who understands and fills a disposable cup with tapwater from the kitchen.  So now I know where I can get water, and it’s not locked – phew!  I also ask about a bite to eat (having missed my main meal), and get a cheese salad whose fresh (though bland) ingredients made it probably the best food I encountered throughout my stay.

Not long after that, things finally start to happen.  They’ve found me “a bed” (seemingly the NHS’s unit of treatment), and a nurse asks me a bunch of questions and fills a form.  They’re admitting me as a patient.  But the bed is in a ward whose atmosphere is not merely hot and stuffy, it’s positively miasmic.  Ten minutes and I’m sweating and panting, so I get up to reclaim my previous limbo-space in preference.  Only now the ward receptionist denies me that space: it has to be cleaned before morning!  I try to escape outside, but find the ward doors locked against me.  Fortunately a nurse is more sympathetic, and finds me another unofficial (and rather nicer) space where I can curl up on a trolley by an open window.  Sometime between four and five I’m seen there by an actual doctor.

After a couple of hours decent sleep they call me to return to my bed for breakfast.  It’s become quite stormy outside, and the wind is sufficient to provide some air flow even in the ward, so it’s now more bearable.  A bowl of cornflakes and a cup of tea, followed by a lot of sitting around doing nothing.  Lunch, and another moment of tension as not a single meat-free (or even non-meat-centred) option is offered!  Eventually they come back and offer cauliflower cheese, which I accept, with a yoghurt for dessert.  It turned out to be something a little different, and reasonably acceptable (for basic institutional food), though the yoghurt was fearsomely sugary.

After lunch they take me off for a scan.  They want to put me in a bloomin’ wheelchair (gotta make work for porters), but I decline, and eventually they let me walk, accompanied by a nurse[*].  Getting back I’m just in time to listen to the last of Ayckbourn’s Norman Conquest plays, which I’ve been enjoying on the radio over the past couple of weeks.  But the storm outside has abated and it’s getting impossibly stuffy again, so once again I go to complain about being detained in such a place.  I badly need a change of clothes, and a charger for the ‘phone, and my toothbrush, etcetera.  Plus, I have strong reasons to want to be back home.  Can’t I just come back by appointment for further tests they want to do?  No chance, we don’t do that.  Well, at least go home to pack a few changes of clothes?  Nope.  But there’s news: they’re moving me to another ward.  The Short Stay Ward should be a bit nicer and more peaceful for me.

It is indeed an improvement.  This time I get a bed by the window, and they’re happy for me to open it.  I wonder about my fellow-inmates, and am immensely encouraged when the patient in the next bed enthusiastically says yes please to opening the window.  So it’s gone from being a place of active torment to a mere place of detention.  I negotiate some time off: I can’t go home, but I can walk around the outside of the hospital for some fresh air and activity, and they’ll ‘phone me if I’m wanted for any more tests.  Outside is a great maze of roads with no green space, and every promising-looking path just brings me to another car park after a few metres, yet it’s still a lot better than being stuck inside doing nothing.  Even food is more relaxed in this ward: there’s a menu, whose options are the same as before, with a few more.  That evening my comfort is further improved by a “patient kit” with toothbrush, soap and towel so I can shower and clean up, and NHS one-size-fits-none pyjamas.

So now my life is the life of the ward, and I’m talking to staff and other patients.  I can see how all the staff are cogs in a huge machine, with their various responses to it and to patients who don’t see themselves as mere widgets on a production line.  Most of them try hard to introduce an element of humanity where possible, and some are very good at it.  I can actually feel marginally useful myself when I’m able to do some small thing for patients less able than myself (of whom there are several).  I have a twinge of regret when I can see I’m not qualified to help when one of the nurses is struggling through a bad headache.

The highlights of the next day are a session with the eye specialist and an MRI scan[*].  And another futile argument about going home, with a glimmer of hope when they tell me just one more test and it’ll be … soon … about ten next morning … aaargh, another night!  I take the plunge and buy Private Eye to see how the eyes will fare[*] and to give myself some entertainment other than just the ‘phone.  It’s too hard to read in the ward light, but next morning I have bright daylight and can read it cover-to-cover.

I’ve been getting into a routine with the catering staff: what do you want for (next meal) … nothing, I’m going home …  Turns out I’m in a pattern the caterers know all-too-well from thousands of patients caught up in this limbo, so they get used to this exchange and generally know best.  I try the only other veggie option, but the so-called curry is utterly disgusting.  And amongst the sweets, only the fresh fruit isn’t smothered in ten times more sugar than a supermarket equivalent.  So that’s a lot of cauliflower&broccoli meals.  They’re smaller than I’d eat at home, but with ***-all physical or mental activity I feel quite full on them.

Next day, ten o’clock passes, the whole morning passes, the whole day passes, my rage and blood pressure are rising.  WTF are they keeping me in for?  This total limbo is truly Kafkaesque, and of course the ward staff I have contact with are not the people who can influence anything.  It’s not just my time, either: they’re supposed to be short of those precious beds, so why are they tying one up with a patient who could perfectly well go home and come back in the morning, or some other time by appointment?  Talking to other patients, I’m far from the only one!

The following morning they finally send me for that test I was expecting.  For a bit of ritual humiliation, a jobsworth porter insists on putting me in a wheelchair: aaargh!  But the test itself is somewhat interesting to a technologist: that’s some impressive medical imaging kit!  I comment on it and mention having worked as a developer on scientific imaging systems, and get into a brief chat that might even have been interesting over a pint if I’d met the man socially.

Afterwards they tell me there’s one more test … aaargh, still in limbo!  But that happens in the early afternoon, so now I can finally await my discharge (the caterers of course know better).  This time, at last, I’m right, though the caterers are also right in that it isn’t until after hospital evening mealtime I’m released.  Four days to the hour after my arrival I wish my ward-mates good luck, and bid farewell to them and to the staff who are around.  It’s a bit late to walk home, so I overcome my embarrassment at clothes so far past their wash-by date and get on the bus home.


[*] Errata are marked thus.  The time I was accompanied by a nurse was when I went to the eye specialist, and that was probably because she put some quite painful stuff in my eyes which might have left me wanting nursing attention.  I had completely forgotten the eye specialist when I first wrote the piece.  There may be more errors, as my memory of the timing and order of some of the tests is unclear.

Verdi Requiem

A week today – Sunday March 22nd – we’re performing the Verdi Requiem at the Guildhall, Plymouth.

This is of course a big work, often described as operatic.  It is deservedly one of the most popular in the choral-orchestral repertoire, and ideally suited to a big orchestra and chorus such as the Plymouth Philharmonic.  Even the non-musical will surely have encountered highlights of it, notably the Dies Irae which is an archetype for terrifying music.  Yet despite all that it’s an easy sing, and – not least – we basses get more than our usual share of the best lines!

This is one of those concerts that is going to be tremendously exciting for performers and audience alike, and I have no hesitation recommending it to readers within evening-out distance of Plymouth.

Saved from Visa

I’ve written before about the Fraudster’s Friend misleadingly named “Verified by Visa”.  Most directly in my post Phished by Visa, though Bullied by Visa perhaps also deserves a mention.

Today I went to place an order with Argos, who I’ve used several times before and who have always – in contrast to some of their competitors – delivered very efficiently.  This time alas the shopping process has become significantly more hassle, and they’ve introduce the VBV cuckoo into the process.  But I was pleased to note that, when I came to the VBV attack, Firefox flagged it up as precisely what it is: an XSS attack, and in the context of secure data (as in creditcard numbers) a serious security issue.

I hope Firefox does that by default, rather than just with my settings.  Though it would be courageous, to take the blame from the unwashed masses who might think VBV serves their interests when it doesn’t work.  Doing the Right Thing against an enemy with ignorance on its side has a very bad history in web browsers, as Microsoft in the late 1990s killed off the opposition by exposing their users to a whole family of “viruses” in a move designed to make correct behaviour a loser in the market (specifically, violation of MIME standards documented since 1992 as security-critical).

Alas, while Firefox saved me from the evil phishing attack, the combination of that and other Argos website trouble pushed me to a thoroughly insecure and less than convenient medium: the telephone.  Bah, Humbug.


Get every new post delivered to your Inbox.

Join 106 other followers