Jason Scott:

Don’t trust the Cloud to safekeep this stuff. Hell yeah, use the Cloud, blow whatever you want into the Cloud. The Internet’s a big copy machine, as they say. Blow copies into the Cloud. But please:

• Don’t blow anything into the Cloud that you don’t have a personal copy of.
• Insult, berate and make fun of any company that offers you something like a “sharing” site that makes you push stuff in that you can’t make copies out of or which you can’t export stuff out of. They will burble about technology issues. They are fucking lying. They might go off further about business models. They are fucking stupid. Make fun of these people, and their shitty little Cloud Cities running on low-grade cooking fat and dreams. They will die and they will take your stuff into the hole. Don’t let them.
• Recognize a Cloud when you see it. Are you paying for these services? No? You are a sucker. You are giving people stuff for free. I pay for Vimeo and I pay for Flickr and a couple other things. This makes me a customer. Neither of these places get my only copy of anything.

In that spirit, I’m going to demonstrate two ways to export your delicious bookmarks into an XML file for safety. As noted previously, this is really a pretty simple thing to do —  you just need to call the URL api.del.icio.us/v1/posts/all and pass in basic auth. For more information on this and everything else in the delicious API, please refer to the documentation.

cURL

cURL is definitely the easiest way to do this. Many web APIs will readily interact with cURL, and delicious’ is especially simple due to the use of basic auth (meaning there’s no need to do a login dance) and the usage of URLs rather than arguments.

curl --user 'username':'password' -o ~/Desktop/delicious.xml 'https://api.del.icio.us/v1/posts/all'

We can turn this into a simple little Bash script:

#!/bin/bash
user='username'
password='password'

curl --user $user:$password -o ~/Desktop/delicious.xml 'https://api.del.icio.us/v1/posts/all'

Ruby

The Ruby form is a little more verbose, but still quite simple. Please note that the following code has largely been borrowed from the nice example here, which unfortunately does not specify a license.

#!/usr/bin/ruby

require 'net/http'
require 'net/https'

user = 'user'
password = 'password'

http = Net::HTTP.new('api.del.icio.us', 443)
http.use_ssl = true
xml = http.start { |http|
  req = Net::HTTP::Get.new('/v1/posts/all', {'User-Agent' => 'Delicious backup'})
  req.basic_auth(user, password)
  http.request(req).body
}

dumpfile = File.new('delicious.xml', 'w')
dumpfile.puts xml
dumpfile.close

Automated backups

Now that you’ve got a script to make an XML backup, why not have it automatically run so you don’t have to remember?

Here’s my crontab, which automatically runs the backup every day at noon:

00 12 * * * /Users/stilist/deliciousbackup.sh

Where deliciousbackup.sh is the Bash script noted above.

Ideas for further development

1. These examples download the entire bookmark collection regardless of whether anything has changed. Checking /posts/update could save on bandwidth.
2. It’s sort of assumed that the service will always be around, and the API call will always succeed. Error handling would be useful.
3. Only a single day’s archive is kept. This means that if something goes wrong with the online copy — accidentally deleting tags, account vandalism, or anything like that — you’ll have a maximum of twenty-four hours to fix it before the backup is overwritten.

Finally

The code here is provided under the MIT license. For a copy, drop by the git repository.

And if you code up any of the improvements noted in the previous section, I’d love to know!

Transcription of Citizen Garden episode 9, ‘Opening Preconditions’.

This one was informative for me because although I was vaguely aware of Ma.gnolia’s plans for their version two release, I didn’t know about the technical aspects — requiring OAuth, self-hosting, and open source code.

The discussion about using OpenID and related ‘open web’ technologies to automatically tell web services where things is quite appealing. This podcast episode is from late August, so maybe work has already begun on implementing such concepts.

But the most interesting part, to my mind, was the idea of using the fact that services tend to come in categories — bookmarks, social networks, and so on — we can export data from those services on a regular schedule and back it up for safekeeping. Then, if we decide to move services (e.g. MySpace to Facebook) or the service goes offline, we’ll have full access and control over our ‘social objects’.

If you think about it, this is really why the open web movement exists. The open web revolves around the idea that if I put information into the system then that data belongs to me. If I fill out a Facebook profile or use Twitter or make a bookmark on delicious, it’s true that somebody owns the system that allows those things — but what use would the system be if I didn’t use it?

So it’s about ownership. But ownership implies freedom — freedom to do whatever I want with my information, since I control who has it and what they can do with it.

This is something that companies offering hosted services dislike. Their business model is generally based on the idea that once you put your information into the system it’s stuck there. Investors see much more revenue potential in a captive audience, because the audience has no choice but to use whatever’s being offered, unimpressive as it may be — a non-technical example would be product placement.

And this is where the whole idea really becomes useful. An important part of the open web is data portability — the ability to easily transfer my information between services. Suppose I’m on Facebook, and I decide I want to try a different social network. With data portability, I’m able to tell the new site about my existing Facebook account and let it import all my friends.

This means that the power situation is changed. I’m no longer stuck using whatever site my friends have chosen. I’m no longer forced to trust that the site I’m on will continue to provide new features and a useful service. Instead, by using a service I’m actually endorsing it, because I’m not obligated to stay there.

It also entices the site’s owner to improve their service. When a site maintains its user base through feature offerings, they’re less able to simply make something good and then let it sit. Innovation is hard, but when it works everybody wins.

But despite all that, I set out to publish a transcript, not an essay. Here we go!

Participants

• Larry Halff
• Scott Kveton
• Chris Messina
• Will Norris

Transcript

LH: Hello!, and welcome to episode nine of Citizen Garden. I’m Larry Halff…

CM: I’m Chris Messina.

SK: I’m Scott Kveton.

WN: And I’m Will Norris.

LH: And we’re here today… I’m freshly back from Seattle after making a big announcement at Gnomedex that Ma.gnolia is being rewritten from the ground up as an open source, downloadable tool. Actually, it’s being broken up into several pieces. I think that’s the more important announcement; there’s a lot of open source publishing platforms out there… blogging tools, there’s even, y’know, I think, a handful of open source social bookmarking systems. A more important thing about what we’re doing is that we’re really trying to pave the way forward with the open web, and part of that is getting these decentralized and federated systems talking to each other. And open source happens to be one way to advance that cause.

CM: Let’s back up here a little bit, ‘cause I’d like to sort of understand better where this idea came from — why you would take this approach when, y’know, you’ve built up a pretty good audience on Ma.gnolia, you’ve got a good user base there, and you’ve done really well so far in supporting a number of these open protocols like OpenID and supporting microformats and OAuth and so forth. but how does open source or open sourcing the platform actually support the open web — and what actually motivated that decision?

LH: Well, I think the primary motivation was seeing that the social aspects of these publishing tools doesn’t really scale when there’s the big single point of failure. and Ma.gnolia has had downtime, Twitter has the infamous ‘fail whale’, Flickr gets massages — and all these things happen, and when it happens you sort of… you lose access to a pretty important piece the flow of your online life. And as they grow, the load that’s caused by exponentially putting out… the exponential effect of putting out all these social connections and publishing and keeping everyone up-to-date just doesn’t… just isn’t really gonna scale in the long term. And I think, y’know, what we’re seeing with Twitter now is nothing like what we’re gonna see with that kind of tool down the road. I mean, hardly anyone really uses Twitter…

SK: Yeah, like a million users maybe.

LH: So I think we really see an important next step is to finding out a way that these things can be pushed out to the edges, yet still have the social functionality of getting everyone talking to one another.

CM: Now one of the things I think, y’know is interesting… so, Scott is with us at Vidoop; he’s also the chair of the OpenID Foundation, and he’s, y’know, sort of one of the champions of decentralizing identity and things like that. I think it’s sort of interesting to think about how OpenID in some ways creates the preconditions for Ma.gnolia going open source, by allowing people essentially to have sort of a cargo horse on which they stack a bunch of things — their photos, their bookmarks, and so forth. Maybe you can — Scott — talk to us a little bit about how you see OpenID maybe as that beginning point that… allows for this type of decentralization that Larry’s talking about, where there are much fewer single points of failures, or at least these concentrations that maybe are creating these pressures on the network.

SK: Okay! Uh, yeah, absolutely! So there’s, y’know, we’re three years into OpenID now, and it was really funny at OSCON, actually — and I think Chris you were on that panel with me, weren’t you?

CM: Possibly, yeah.

SK: Leah Culver got up…

CM: Oh, right.

SK: … started just dropping f-bombs about how she just thought OpenID was dumb, she’d never use it, da da da da. And y’know, in reality, it’s… it is a URL-based system, and that’s just… users have not, y’know, grokked that. But I think what we’re starting to realize is the value is proving a potential service endpoint, which means nothing to users.

CM: [Nor necessarily?] should it…

SK: Right, absolutely.

CM: For now.

SK: … but for developers it’s extremely important. So, y’know, some of the challenges that we’ve been facing in the OpenID community are around security and usability. If we can make it easier for people to identify themselves — whether with an email address or identity in the browser — then they can get in, prove that they are some end point without having to know what that endpoint is, and then start to put their data there. and then we start that… that lays the groundwork for things like y’know, lower-case data portability and the ability to, y’know, have more control over who provides your… or manages your data.

CM: Yeah, I’ve been thinking about these things lately, and it… it’s interesting to reflect on what assumptions we bring to these problems, especially around sort of, as you talked about, the developer part of the equation, where you’re starting to think about and understanding well, if a developer starts to tackle a problem, trying to build a new service on the social web, what assumptions can they make today that they couldn’t make before? And previously maybe you made an assumption that people would have an email address before; okay, great, then you could send them a password or a token, and they can prove received that token — that’s a way of confirming… that’s a durable identifier, but you can’t really attach services to it. You can’t actually look up that email address and ask it, y’know, ‘well where is this person’s photo store’, y’know, ‘where is this…’ — there’s no directory for that. Using URLs as identifiers sort of helps to at least make that situation a little bit better. So I’m actually interested in hearing from Will, ‘cause Will and I work on the DiSo project together. How do you think the ability to use unique URLs to identify people — which then have services offered at the end of them — changes what you can sort of take for granted? What are the building blocks now, that when you approach building a new application, you’re like oh, well they probably have identity, they probably have, y’know, some service that they’re gonna use, we’re just gonna throw it all together and make it happen?

WN: Yeah, right. I mean, we’ve been trying to address those exact problems of ‘how can we build that infrastructure so that you can take advantage or take it for granted?’. And like you said, y’know, you can’t attach these services to email addresses. Recently, I co-authored a spec — and we’re now using this with a lot of our stuff — called ‘EAUT’, which is e.a.u.t. It’s Email Address to URL Transformation. It basically allows… it’s a really standard way of taking that email address and talking to whichever email provider it is that provides that, and say, y’know, ‘how can I turn this into a URL that I can then go and try to find these kinds of services on that URL?’. So y’know, Yahoo! can host their own thing, Google and all this, and then we also have this fallback service. But the idea is that, so, we can use an identifier that users are familiar with, and comfortable with, and use everyday, and they love — it’s their email address — but still be able to get these additional kinds of things. So, y’know, when I go in to whatever it is, I can give my email address, it can be converted into an OpenID; this application can then go and look at my OpenID and say ‘hey, y’know, we need to publish a bookmark, where should I do that?’ And, y’know, I could be doing that on Ma.gnolia proper, or if, y’know, I have my own Ma.gnolia instance running, with the new open source version, or if I’ve, y’know, maybe… maybe Vidoop, y’know, we set up our own, for our employees, I can say y’know what, that’s where I do mine. And so all I have to do is present my identifier, and, y’know, magic happens, basically, and this consumer can push that bookmark out to wherever it is that that I store them.

LH: Right. I think it’s important, it’s like that’s another precondition we haven’t explicitly mentioned. It’s like, we have the… we know identity exists, and we also, with the unsung but super-important spec XRDS Simple is really key to making that happen, because at least for programmers, it’s like, they can go to the end of your OpenID, and they can say ‘well, this is someone’s identity URL’, and they can get a whole bunch of links off of that. But they’re really meaningless until we have a way of saying ‘what do each of these links do?’, and so I think, I just… point that out as a key piece to that, which isn’t just having the thing, but also having the mechanism to say ‘what can we do with these different things?’.

CM: To put that another way, it’s kind of like when you go to someone’s blog and they have a sidebar that lists all their other profiles. What we’re trying to do, I think — and this is sort of what this discovery protocol is all about — is taking that list of URLs and making them make sense to computers, essentially. So that when it sees a Twitter icon, y’know, more or less, it’s like ‘oh, that’s a status update service’, or ‘that’s a microblogging service, therefore I can post messages to it, if I’m authorized to do so’. Very similarly, if you see a little Flickr icon, or if you see a YouTube icon, those are different services that someone might use that actually have APIs that you can talk to. So if you can advertise those URLs through this discovery/specifications at the end of an OpenID identifier, that’s where the magic starts to happen. So, what I was actually interested in hearing you two guys talk about a little bit — and recently, y’know, you’re wearing the WordCamp shirt, and you were at WordCamp, will, and you talked about OAuth for WordPress. and this is, I think, very interesting, because we’re at a point where WordPress currently does not support OAuth. Most of its transactions are done with the standard username and password, which means that if you wanna, let’s say, blog from your iGoogle home page, you’re gonna have to enter in your WordPress username and password into iGoogle. Well that’s great, except when you start doing this across the web and so on, and that’s the password anti-pattern. now on the other side, we have Ma.gnolia, which already supports OAuth in the platform. And I’m interested sort of in hearing you guys talk a little bit about the pros and cons and the challenges of retrofitting OAuth and authorization-based permissioning into a platform like WordPress, whereas Ma.gnolia already has that — Ma.gnolia open is gonna come out supporting that from the get-go — what does that mean for people building on these different platforms? How does that actually improve the situation?

LH: I think it improves the situation that enables a lot more seamless experience for the end user that… I think combining OpenID with discovery with something like OAuth is, y’know… this is a whole lot of hot air, so I’m ashamed of the words that are about to leave my mouth, but the browser of the future…

CM: Uh-oh.

LH: There, I did it.

?: He did it! He did it!

LH: … will be like…

SK: Tshirt’s already been ordered.

LH: The browser of the future…

CM: dot com!

SK: That’s right. Do we have that yet?

LH: … when I go to save a bookmark, instead of it saving in by browser — it will have known my OpenID already, because when I launch my browser and it’s first setup, it says ‘what is your OpenID?’ — it will have verified that, it will have discovered my online bookmarking service, and it will know where that lives, and as part of that process it will have authorized access to my bookmarking service account though OAuth, and I will have said ‘yes, this browser allowed to post things to my bookmarking service’. And so ‘save bookmark’, it will be seamlessly integrated to me. And that really is the end-user benefit for all this, despite all the horrible, geeky, completely incomprehensible nerdiness and the ongoing usability issues with OpenID, which Ma.gnolia open has cracked open again.

CM: Yes.

LH: If you wanna participate in a great little thread about OpenID usability, go to the magnolia-2-discuss group on Google.

SK: Yeah, y’know, if there’s one thing I’ve learned over the last three years of the OpenID stuff, is it doesn’t matter how open it is, if it’s not usable, it’s broken. And so that means, y’know, I think usability has to come first, and I think we have to break some things around the openness of it to get it right first. And we’re seeing, y’know, Facebook — we were talking about this today. Y’know, what Facebook is doing sort of, y’know, in the eyes of a lot of folks who are very open-centric — which doesn’t really make sense, but anyway — they see that as awful, because they…

CM: Whoa, wait; be more specific about what Facebook’s doing.

SK: Well, they’re embedding an <iframe>, effectively, on other sites — and this is effectively what Google is doing as well.

CM: That’s right, that’s right.

SK: And, y’know, that’s all well and good for the sites themselves — they don’t actually get the access to the user information — but, they can get more people and pageviews, which could be really important to them. but from an open perspective, it’s not that open, to be able to do that. And… what’s the other thing I was gonna say? God, this coffee really does — oh my god! sorry.

LH: These guys bought you Blue Bbottle before?

WN? Sponsored by Blue Bottle…

LH: I am writing without Blue Bottle here.

CM: It’s a plug for the [something]. Oh my god.

SK: It’s good stuff.

LH: But, so I mean… so I think we were getting… we were also talking about the whole convergence to this stuff, and what’s, like, what’s… the work you’re doing with WordPress, and like, where’s… do you see that headed in a similar direction? Is WordPress thinking about this?

WN: In a similar direction as..?

LH: As sort of, like, end-to-end integration…

CM: Well the interesting thing is that Weave — which is a Mozilla Labs project — is kind of in that direction, where it actually does kind of what you described, Larry, in allowing you to kind of sign in with some accounts, through OAuth actually authorize the browser to both publish your bookmarks and download ones that you’ve already saved — very much like a MobileMe for the rest of us, in a sense, for those of us who are not gonna pay Apple or whatever to do so. And all of… the entire sort of service stack of MobileMe could be more or less built on open technologies. It’s interesting, though, to think about what it would mean for someone like WordPress, or even someone like Drupals of the world and so on to really embrace some of these technologies, and to look at the opportunity that the browser, y’know, deep browser integration and web service access and offline storage, to some degree, would offer. And so I guess the question for Will is sort of around, y’know, what would OAuth mean to the WordPress platform? How would that accelerate the development of things, how might that make WordPress a different type of integral platform for publishing all sorts of different services on the web, perhaps?

WN: Well I guess, I mean… the most immediate use case, I think, that we’re gonna see with getting OAuth into WordPress is just allowing whatever service it is to publish to WordPress without needing the user’s credentials. So, this could be something like the WordPress iPhone app, this could be MarsEdit, it could be… Flickr already has a way where you can push your photos directly from Flickr into your WordPress blog…

CM: Well Ma.gnolia 2 actually offers publishing your bookmarks to your blog.

WN: Oh, does it?

CM: But right now, I believe, you have to take… it’s the password anti-pattern.

LH: Yeah. It’s the password… nobody… yeah. The major blogging platforms except for Blogger — which we don’t support because they don’t use the MetaWeblog API — use OAuth authentication. And I mean, that’s great for Google [something].

CM: And Movable Type is going to be supporting OAuth in the next release, I believe.

SK: It’s already out.

LH: It’s already out.

SK: The libraries are there, they’re in the core release… four, two, whatever it is.

CM: But still, we’re still at a point where we need that deeper integration, but…

SK: Well, yeah, and just kinda playing off that a little more is that a lot of people that I’ve seen are using WordPress as kinda more of a persistent storage of their social objects. Y’know, with Ma.gnolia, y’know, you have… there’s things going on with Ma.gnolia, but, y’know, Ma.gnolia might go away tomorrow, or I might wanna move to some other platform, so I wanna have a copy within my own control. So, y’know, I do a nightly pull or push or whatever to my blog. People do that with their Twitters — er, their tweets, y’know, they’ll do a day’s worth of tweets as a blog post.

?: My tweets are very important.

SK: Well yeah, absolutely, that’s how people feel. And whatever that…

LH: Live coverage of the Olympics…

?: [something] my addiction.

SK: So yeah, I mean… just simply using WordPress as kind of a persistent storage of these objects that are within the individual user’s control. And in order to make all that stuff happen in a secure way, yeah, absolutely, you’re gonna need a secure mechanism for pushing that in, and that’s gonna be OAuth at some point, once we get that built.

CM: And I think… well I dunno, we probably should wrap up pretty soon…

LH: I think we’re heading towards that.

CM: Yeah, well… so I guess actually we could close on some final thoughts, since this has sorta been a whirlwind discussion… and there’s much more, obviously, we could talk about. What it sounds like you’re talking about — I really like the way you framed it in terms of kind of your store for social objects, your generic store, is that increasingly we’re gonna have specific tools that do a good job at storing different types of social objects and providing metadata around those objects. So we’ll have Ma.gnolia, the Ma.gnolia bookmarks is like the WordPress of bookmarks in a sense, so you might use your self-hosted WordPress — I’m sorry, Ma.gnolia install — to host bookmarking-type things, which have certain screenshots of the webpages, maybe some tags, so on and so forth…

LH: And have access to all that’s going on out there.

CM: Right. And maybe you’ll also be able to push those bookmarks out, and also pull things in via that type of channel, because again, that channel is designed specifically for those types of objects. Then you have your Flickr, which might be a better photosharing application, or maybe you wanna use Facebook to view photos. I mean, who knows. Whatever the case is, moving these objects around into different web applications seems to be where this is gonna go, and being able to push the data around fairly [in a] fairly straightforward way using OAuth to control sort of who has access to read/write, that’s important, and then coming up with the standard protocols so that each endpoint kind of understands what kind of data is being pushed around is also a matter of import, I think, as well. So it’s really interesting to think about how we can actually move to real cloud computing using these types of protocols. So that was a longer closing from me, so what do you think?

SK: Yeah, I think… I think as we move closer and closer to having… putting users in control of their data — and I actually really like that term ‘social objects’ — because to me social networking actually isn’t something you do on the internet, it’s just a feature? And especially when you apply it to things like bookmarking services or photosharing sites, I wanna be able to bring my social network along with me. It should just be a foregone conclusion. And so to me, the work that we’ve all been doing has been headed in the direction. And, y’know… good stuff.

WN: Yeah, me too. Plus one. Plus one!

SK: Plus one for me.

LH: I think for me, it’s like, what we’re gonna be seeing next, since — is that mandatory, do you have to close with ‘what’s next’?

CM: No.

SK: Absolutely!

LH: But I’m gonna do it. What we’re gonna be seeing next is — and Ma.gnolia isn’t the first — but we’re gonnna be seeing these kind[s] of services becoming more decentralized, which means… which means creating another problem. But the ‘more decentralized’ means more reliability, more control, more adaptability to individuals’ needs. But that removes a lot of social functionality, removes a lot of community, removes a lot of interaction. So we’re gonna be seeing that problem solved. We already know how to decentralize; we do it with blogs. It’s there. But we’re gonna be seeing the federation problem being solved over the next few months. And we’re gonna see how we can bring those together in more of a ‘small pieces’ type solution [something] social network.

SK: We’re gonna solve the problem in six months?

LH: Yeah. Yeah.

SK: All right, let’s go.

CM: We’ll be there.

LH: High five.

?: Yeah!

CM: So just one more plug for… it’s Ma.gnolia.org is where you’re gonna find this stuff, and it’s ma-dot-gnolia-dot-org, that’s where you can find out more about the announcement, the m2 — as it’s being called — charter, and…

LH: And the Google group.

CM: Yeah.

LH: Come and join and contribute to the discussion.

CM: There you go.