‘Link to the rest’: the 12-year journey to 1,000 days of @TheOverspill links

15 min readFeb 11, 2019

A bookshelf overspill. CC-BY photo by Greg Pye.

So today, Monday 11 February 2019, is the 1,000th edition of “Start Up”, a daily list of links plus brief commentary, on my Overspill blog. (Also available on email. Open it in a new tab and keep reading.) Does that sound a lot? The first one in earnest was in October 2014, though the very first was in July 2014. At five per week, that’s 200 weeks — around four years, before allowing for holidays.

Curating a daily collection of interesting links with commentary is hardly a new idea. But the evolution of how we got here seems relevant, because it takes in the evolution of news over the past couple of decades, what it means for journalism, a sprinkling of life and death on the web, plus programming and cross-platform evolution.

‘Link to the rest’

To reach The Overspill’s Start Up, we have to walk a long road, via the Guardian’s technology and media sections in the mid-2000s. I really first noticed the idea of simply linking to interesting stuff being done by John Gruber with Daring Fireball, back in the early 2000s while I was working at The Independent as a science and technology editor. That was also a time when there was a sudden growth in “news” sites which were essentially collective blogs written in reverse chronological order.

I joined the Guardian at the end of 2005, and was very interested by what it was doing around blogging: the technology section had blogs both for technology and for games, and the blogs’ content was a lot more free-form — looser, more personally voiced — than the formal pieces that were being written for the physical paper, or even news pieces for the web site. It was also a time of huge experimentation at the Guardian, where the then editor Alan Rusbridger had identified the commercial threat of digital: many more readers than print, but far less monetisation per reader. The thinking thus went that if you could wallpaper the place with content that people might want to read, then you could perhaps get enough readers to fill in the “green blob”, as he called the gap between the revenues you got when tons of people bought the paper, and the much smaller revenues you got from web readers.

I was mindful of a couple of things. There was Clay Shirky’s blogpost from, I think, 1995 titled “Help, The Price Of Information Has Fallen And It Can’t Get Up”. In it, he pointed out the basic economics that “prices fall when supply outstrips demand, even if both are rising. [This] describes the network perfectly, since the Web is growing even faster than the number of new users.” (You can tell it must be 1995 because he capitalises “Web”, doesn’t mention Google, but does mention a search engine called Lycos, and I bet a proportion of you weren’t even born when that was a thing.)

He pointed out:

To take newspapers as a test case, there is a delicate equilibrium between profitibility and geography in the newspaper business. Most newspapers determine what regions they cover by finding (whether theoretically or experiemntally) the geographic perimeter where the cost of trucking the newspaper outweighs the willingess of the residents to pay for it. Over the decades, the US has settled into a patchwork of abutting borders of local and regional newspapers.
The Internet destroys any cost associated with geographic distribution, which means that even though each individual paper can now reach a much wider theoretical audience, the competition also increases for all papers by orders of magnitude. This much increased competition means that anyone who can figure out how to deliver a product to the consumer for free (usually by paying the writers and producers from advertising revenues instead of direct user fees, as network television does) will have a huge advantage over its competitors.

In other words: newspapers that don’t do something special compared to competitors are screwed.

I’d also noticed how what we might call the velocity of content was changing.

Faster, pussycat! Kill! Kill!

When I first started at a national newspaper in 1995, the aim was to write one well-sourced and researched story in a day. This was entirely for print, in a world before people had dialup internet on their desk, let alone high-speed broadband in their pocket. (At least they had mobile phones. You have no idea how difficult it used to be to get hold of people before mobile phones.)

Within five years, everyone had broadband on their desk, and now there was a lot more information coming in, and a lot more places to look. One of the moments I recall was managing to find the patent filings for the Segway just a few minutes before the deadline for delivering the piece at 5pm, which could then be used to illustrate it. That was January 2001. (Weirdly, that piece isn’t on The Independent’s own site, but is on its Irish sibling paper’s.)

Then everyone had web sites, and there were web sites which simply pulled together lists of things that were on other web sites. Everything became connected.

That also meant that the number of stories you could potentially write in a day — check and research for yourself, write, publish — went up dramatically. From picking one story a day for print, suddenly you could write three per day, having picked from 10 potential ones, because the checking process was easier.

As time wore on, you could still write three, perhaps five stories in a day (there’s a ceiling on how many stories you can reasonably research and write if you’re trying to actually add some value, not just rehash it). But there were now 20, then 30, potential stories. Often some of those stories were versions of some of the other. The velocity of news had increased enormously. And this was only 2006 or so, well before Twitter, the news junkie’s crack, had got into its stride.

I was also mindful of Jeff Jarvis’s February 2007 advice, or perhaps dictum, for news sites: “cover what you do best, and link to the rest.” It seemed to me this was and still is good advice: in a world where available content is expanding all the time (per Shirky, above), you’re essentially going to be competing against everyone online if you try to catch up on every story and rewrite it.

In the online world it’s far better, I thought, to add in what you do best — which in the case of national news organisations ought to be “know more about the topic, or know the people who know more about the topic, or have previously written about the topic, so you can add context and understanding” — and link to all the rest of the content out there that you didn’t have a hope of getting around to. Just to point out that it’s worth noting, but you’re not going to rewrite it because there are better things to do with your time.

Enter the newsbucket

As far as I know, the idea of doing a daily list of technology/media news links came from Jemima Kiss, who joined The Guardian’s media site and ran, among other things, the PDA blog, which was set up to be “The Guardian’s blog on digital content, digital culture, and technology meets media”. (Don’t blame me for the description’s grammar.) She called hers the “newsbucket” — the place where you’ll find all the news you’re looking for.

The first “Newsbucket” was on 25 September 2007. That first post’s contents included the launch of Amazon’s music download service, the simultaneous shuttering of Virgin Digital’s online music store, and a warning that “even a video episode with 20m viewers has to cost under £200,000 to make any profit” — that warning coming from, deep breath, MySpace. In other words, some things you might have missed if you’d only stayed glued to the Guardian site, but which equally didn’t necessarily merit a full writeup. Ten links. Short and sweet.

At the time, Jemima collected the links on Del.icio.us — she would read through her newsreader on the train travelling in and bookmark things — and then, I think, hand-crafted the page that then went into the CMS (content management system). Oh my. Hand-crafting. When I learnt about this, I helped — I hope — by writing an Applescript (because the office was on Macs) that would query the Delicious API, find the newest links, and format their headline, URL and comment into a form that could be added straight into the CMS. It turned a job that could take half an hour into one that took 5 seconds. (And such fun was had writing a subroutine to change The American Style Of Noisily Capitalised Headlines to The English style of lower case headlines while allowing for company names like Intel and, uh, Apple?)

Early riser

We’d also begun to like the idea of something that would be available really early in the day, so that people hitting the site early wouldn’t find that the freshest thing was last night’s content. Bobbie Johnson, then the Guardian’s San Francisco-based technology writer, had begun writing a “Breakfast Briefing” (‘PC industry stabilising, says Dell’ — in 2009) which would go live at about 6.30 in the UK, and include things that had happened in the afternoon on the west coast and so been missed by us in London due to trivial oversights such as sleep.

When Bobbie moved on (to create Matter — later bought by Medium; isn’t life weird?), I thought it would be good to have something similar to the Breakfast Briefing to greet readers. I began a Technology version of the Newsbucket, but tried to make it more technology-focussed rather than media. It was however inevitable they’d overlap. The first seems to be on 29 March 2010, and the first link is to a Newsweek article by Dan Lyons (aka Fake Steve Jobs, as he’d recently been found to be), explaining “Why the iPad Will Change Everything”.

Amazingly, the link still works, where Lyons declares:

“The iPad could eventually become your TV, your newspaper, and your bookshelf. Pretty soon, Apple might even become your cable company — sort of — by selling subscriptions, via iTunes, to individual shows or channels. Say you’re reading the latest Henning Mankell on your iPad. While you’re sitting there with it in your lap, why not check your e-mail or flip on an episode of The Office?”

(Or you could flick over to Twitter. Or, indeed, write this post.) Trip Hawkins, founder of Electronic Arts and 3DO, predicted that within a decade there would be a billion tablets computers in use, but “I’m probably being conservative”. Also on offer was the idea there would in time be “tablets on plastic sheets that you could roll up or fold out like a map”. That was predicted for “the second term of the Chelsea Clinton administration”, so there’s a little leeway for that to happen, since foldable screens are just beginning to appear. (Though, as Lyons pointed out, it didn’t run Flash. Obviously doomed.)

A little more organisation kicked in by April 2011, and we began using a “guardiantech” account on Delicious, so that multiple people could add links. (This didn’t happen. What tends to happen in systems like this is that one person adds links, and everyone else doesn’t. That’s fine; posts like this benefit from an authentic single voice. If enough people like the voice, it will do OK. If they don’t, it will sink. That’s life.)

Over time, the PDA blog on which Jemima’s Newsbucket appeared was folded into the wider remit of the Technology blog. The last PDA Newsbucket I can find is from 15 September 2011. You can see that it has changed a lot in design: rather than the description, all you get is the headline and >> and a hyperlink to the story on the site, with the site’s name.

Infrastructure week/month/aeon

Meantime, the internet infrastructure wasn’t looking so clever. Delicious began to look dubious as Yahoo began floundering. In December 2010 a leaked slide from inside Yahoo indicated that Delicious would be “sunsetted”. Uh-oh. Yahoo clarified that this meant sold, not shut. We weren’t reassured. The site still ran under Yahoo until September 2011, and then launched in a New!! Beta!! which was a royal pain.

So we took the very easy decision to copy all the old links to Pinboard, because it offered the same API as Delicious (hence minimal rewriting of scripts) and a fixed one-off price if all you wanted to do was bookmark, though the later you joined the higher the price became. (It’s now shifted to a subscription basis, though existing one-off payment accounts are grandfathered). Pinboard’s strength was and is that its owner doesn’t harbour dreams of turning it into a video news site or chat forum. Maciej Cieglowski, who is in charge of it, just wanted to run a really good bookmarking site. And so we watched Delicious, owned by something mysterious called AVOS Systems — which turned out to be the guys who originally came up with YouTube and sold it to Google — sail off into a really weird sunset. Spoiler: Delicious went around the block a couple of times and then Pinboard bought it for a song.

This led to the situation where there was a “Technology newsbucket” running each morning, and a PDA Newsbucket and also a Gamesblog newsbucket. That’s a lot of buckets. It is fun to dip in and see what people were certain about in 2010: “Where are the Android killer apps?” (Daring Fireball) — which is still a valid question — and “The profit motive will kill off seven papers by 2020” (Guardian). Plus all those articles about Chatroulette. Remember Chatroulette? GuardianRoulette? StreetViewRoulette?

By now the links were coming from Google Reader . The publication timing of the PDA Newsbucket would vary because Jemima had to come in on the train, adding links en route, and then set up the blogpost.

And so the rhythm of the day worked out: spot an interesting link, bookmark it with some commentary (I’d written a script that would blockquote excerpts), then run the script last thing at night, or first thing in the morning, to aggregate the links, create the post, find a picture, post. The hardest part often was — and still is — finding a picture to go with what seems the most interesting or intriguing story. (Also, I overuse “interesting” and “intriguing” when discussing links.) That was especially true after Those Who Decide These Things ruled that we couldn’t use CC-licensed pictures from Flickr but had to use photos to which it was certain we had the rights. This seemed to me a misunderstanding of what “CC-licensed for commercial use” means, but they were the ones in charge.

And so things went until October 2014, when I left the Guardian.

Curating a daily set of links for four years is something you get quite used to. There’s a rhythm to it. More important, it gets you to read a lot — you have to stay abreast of what’s happening. So I kept doing it. And with the invaluable help of Tom Davenport, who helped me set up the Mailchimp integration (RSS feed -> email every day!), there are now twice as many readers on email as there are on the web.

The aim is much the same: point to stuff that’s worth knowing about. Fill in the edges of what you can generally get at the big sites. Discover the best analysis, the earliest warnings, the undercurrents that will become riptides that change how we think about something. I do enjoy occasionally seeing articles that I’m fairly sure have been prompted by something in The Overspill. But of course you never know. You just do it because it’s worth doing.

Script for the day: without code, you’re lost

Scripts — programs that aren’t compiled apps — were very important. Basically, if you can’t script these things, don’t even think about doing them, because you’ll make yourself weep at the prospect of having to do it day after day. By contrast, there’s a delight in simply hitting a button and having it all happen automagically. By the end at the Guardian, I would run one script and it would query Pinboard, format the links, prepare the post, and then fill the CMS with the body, story tags and time of launch (0630 the next day), which took about 15 seconds. (Crucial to this was an app called Fake, which was scriptable.) All I needed to add them was the picture and the headline.

Through its life, the script did the same thing — query the API service, format the headline, URL and comments— but it had to work on multiple platforms. Sometimes people were doing it on their personal Mac. Sometimes on a Mac that wasn’t theirs. That meant I had to rewrite the script from Applescript to PHP on a private web page, at least to collect the links and format the headlines, URLs and comments. (It’s much easier to do this stuff in PHP than Applescript, and I came to PHP after Applescript; also, it’s easier to do in Python than PHP, but harder to set up a web server with Python.)

Lately, I’ve been doing the process on an iPad. So overall that has meant I’ve had to write the script, or a version, successively in Applescript, PHP and most recently Python linked to Shortcuts. (That version, on the iPad, puts in a picture I’ve chosen, formats the links, and sets up the Wordpress post with the publication time. Thus it’s faintly better than the Guardian setup.)

Being able to write scripts was a lifesaver for my sort of journalism, where analysing data often mattered. Querying lots of webpages, pulling things together, drawing graphs — scripts helped. I’d still say “learn to script” to any journalist looking to get in to the game, because it gives you something extra that many won’t have. Certainly I wouldn’t have been able to get the developer time to write a crappy little script, and wouldn’t have been able to transfer that when I left.

Death and the web

As noted above, services couldn’t be relied on. Delicious was taken for a ride up to the mountains by Yahoo. Pinboard, thankfully, has been reliable, because Maciej just wants to run a successful, profitable service. Google Reader came and went. Sometimes, other events intruded: sometimes I’d want to send a link for later contemplation, and on mobile I used to save those to Instapaper . But it then had a conniption over the introduction of GDPR in Europe which lasted for months. So I switched to Apple’s Reading List instead. Sayonara Instapaper, Google Reader, Delicious.

The obvious lesson: you can’t be sure that free services won’t go away, so don’t build something that relies completely on them — have a Plan B. Paid services, on the other hand, can be more reliable, but you need to have an idea of the ambitions or otherwise of those running them.

‘Where do you get your stories?’

Where do the stories come from? All over. Generally from Twitter, but also from other sites that I look at from time to time. In general, there’s too much information rather than too little, as I said. But even with the accelerating velocity of news, much of the stuff out there is either a direct lift of something else (in which case I try to go to the original source) or is more like a press release. Despite everything, the amount of news one needs to pay attention to has remained about the same for the past 20 years. All that’s changed is the visibility of all the other stuff — the flotsam and jetsam.

Trends

I’ve collected somewhere north of 10,000 links over the course of four and a bit years. The broad trends I’ve noticed are:

concerns about AI taking over everyone’s jobs have expanded, then receded.
smartphones have gone from boom business to stable to slightly contracting, with the consequent squeeze on smaller companies.
smart speakers went from being the New Hotness to a source of concern about invasion of privacy.
the internet of things has just remained a complete mess, with business models that conflict with security.
AI has gone from overblown to “will this really do anything useful?” Meanwhile, I think it is going to do something useful. But we might not notice it.
Smartphone reviews have become a form of squeezing blood from a stone. There’s so little difference between them.
Adblocking looked like a giant threat to lots of companies. But it seems to have stopped growing.
Bitcoin really has become a place for crooks and idiots.
and many more which I’ve surely forgotten.

The most popular links, in terms of clickthroughs, haven’t been about technology. They’ve been about

I don’t take that to mean that I shouldn’t do links about tech anymore; rather, that people like stuff that really surprises them, and we tend to be most surprised by things outside our general area of interest.

That might not seem like a lot of trends. But it feels as though, post-smartphone boom, we’re in a period of stasis. Augmented Reality glasses? Google tried them. Failed except for niche commercial use. Microsoft tried them. Failed except for niche commercial use. Snap tried them. Failed. Smart speakers are nice, but they’re essentially Bluetooth speakers you don’t have to control from your phone, until you do.

So what’s the next big thing, given that the days when the iPad was the Next Big Thing seem far off, and wearables haven’t yet quite taken the world by storm? (By the way, we’re now at 1.8bn tablets sold cumulatively since 2010, but it’s hard to say how many are in use. Trip Hawkins might yet be proven correct.)

Future trends? AI is hugely important, but we’ll hardly notice how it changes things. In that sense, we’ll be like the proverbial frog in the warming pot. The other key trend is climate change — which really is the frog in the warming pot, along with the rest of humanity and the ecosystems we depend on. Can one prevent the other? I hope so. Stay tuned.

If you enjoyed this, you might like my daily list of links and brief commentary at The Overspill blog, also available as an email (signup and confirmation required — free, no ads) or by following @TheOverspill on Twitter. I’m @charlesarthur on Twitter.