Transcription of Citizen Garden episode 11, ‘Whither Magnolia’.

As you may or may not be aware, the Ma.gnolia bookmarking service recently lost its entire database.

I was not personally affected by this loss, as I instead use delicious and back up my bookmarks daily. I had briefly tried Ma.gnolia; after a lengthy wait while it processed my bookmark collection, I soon decided that the system simply didn’t allow for the things I wanted to do.

But even if I didn’t lose anything, a lot of people did. And if delicious were to disappear, I might still have my data, but I wouldn’t have a way to use it (immediately, anyhow; there are open-source importers). So this is still a good thing to think about.

Although Chris Messina doesn’t seem overly concerned with the loss of his bookmarks, I make heavy use of my bookmark archive — collecting things like article series, free-culture content, free music, and a variety of other purposes. For me, it doesn’t have a half-life of twenty-four hours; I bookmark so that I can quickly re-find things that have interested me. I’m willing to grant that I’m an outlier; the number of tags I use per bookmark likely ensures that anyways.

But even if bookmarking is done on a very short-term basis, it’s useful, as the podcast points out, for things like generating recommendations. A major trend in my feed subscription habits is that I love sites which point out things I’d never see otherwise. As Dave Winer says, People come back to places that send them away. Although I didn’t use Ma.gnolia in its pre-crash form, I’d be very interested in one that tried to give me a list of interesting links. I’ve lately begun skimming the front pages of Digg and Reddit several times a day — which, while useful, also means I have to see a lot of things that I really don’t care about. Links recommended by a computer aren’t quite on the level of links recommended by people I trust to be interesting, but it can be very close.

Regardless of whether Ma.gnolia ever appeals to me personally, I hope it comes back stronger than it was. Competition is good, and the service has a chance to move things in interesting new directions.

On to the transcript.

Participants

• Larry Halff
• Chris Messina

Transcript

LH: Hello, and welcome to episode eleven of the Citizen Garden podcast. We’re actually coming to you today via video as well. I’m Larry Halff.

CM: I’m Chris Messina.

LH: And today we are going to talk about what happened with Ma.gnolia.

CM: Yeah. Which I guess is, for many people, not that funny, but uh, it’s fairly, I guess, sort of a momentous thing, and of course you being the news creates an interesting opportunity for us, I guess, to both talk about what happened, and for you to sort of explain maybe the situation as it occurred, what’s happened since, what you’ve done sort of in response, and maybe some lessons learned here. So maybe you wanna give us some background on what actually happened.

LH: Uh, so, I still don’t have all the details on what happened; still working with a? data recovery company and hope to know more when I hear back from them, but: what seemed to me to have happened was we had some file system corruption and our very large database file got corrupt and…

CM: What size database file are we talking here?

LH: It was approaching half a terabyte of everything together, and…

CM: Is this MySQL, or…

LH: Yeah, MySQL; MySQL 5.

CM: Okay.

LH: And… I think this had been somewhat of an ongoing issue, but everything was running even though this was going on. And eventually it stopped running, and the site went down. It just no longer worked. And because of this, our not-so-awesome backup system also failed, because it was not able to properly back up the data from MySQL.

CM: Is that because of the size, or I mean… so what, maybe you can talk a little bit about what you know what happened with the backup.

LH: So what happened with the backup was it was just trying to back up bad data, so whatever the backup produced was not usable either. It was just giving a file sync over a Firewire network to a different machine. So, in this case, because we didn’t have a good sort of integrity-oriented backup system, it failed.

CM: Now, had you ever done tests or anything like that to see..?

LH: Nope. Had not purposely failed the database to see what would happen.

CM: I see, so…

LH: Which would… which is one of those lessons learned, which is: test your backups, test your backup system. I don’t know that a test would have caught this sort of thing, but it’s something we should have done. And another lesson learned would be: figure out your backup, figure out a good versioned backup system early on. Or actually the real lesson learned is if you’re a startup, don’t do your own IT at all, which is… And I think three years ago, it was less of an eff — three, four years ago it was less of an option. Ma.gnolia, I really started on Ma.gnolia four years ago, and we were running the beta over three years ago. And there was not… there was no sort of cloud computing at that time. So it was the, you know, the option was really bad hosting, especially for Rails applications, hosting that…

CM: It almost didn’t exist back then.

LH: I knew wouldn’t scale. Or do-it-yourself. And sort of in the process of developing Ma.gnolia, infrastructure always sort of took a back seat. And along the way we suffered because of that; y’know, I’d say in about 02006 we definitely attrition from the service because of speed and reliability.

CM: Performance.

LH: Yeah, the performance, the site would slow down. But because you… because we were developing Ma.gnolia specifically for the environment it was deployed in, there was… there is a huge tax to sort of moving to a completely new environment. We have all sorts of dependencies, all sorts of stuff that we required in our specific environment.

CM: Now, I mean, maybe you could talk a little about the actual infrastructure, y’know, the environment you had set up, from a hardware perspective. Because I think one of the things that, y’know, most people probably have no insight into, y’know — unlike your Mac you can’t go to the little Apple, y’know, and choose ‘tell me about this Mac’, and get the specs.

LH: Right.

CM: Y’know, for web apps. And maybe you can talk about, you know, the actual system that you were running Ma.gnolia off of.

LH: So we were running Ma.gnolia on… the database and the backup were on a couple of Xserves, and then we had about four minis…

CM: Mac minis.

LH: Mac minis. ? Mac minis that were running as frontend web servers. So it was a very small setup, and… I mean, interestingly, y’know, with a pretty good Xserve as the main database server, it ran pretty well.

CM: And you weren’t doing anything like RAID or anything with it, it was just Firewire backup.

LH: The server was RAID. Its disk was RAID, so that’s one of the things we’re looking at. But it was a software RAID, so if it’s a filesystem problem then… that’s not gonna do any good because the the errors were RAIDed as well.

CM: So let’s talk a little bit about, I mean, the reaction, y’know, to this so far. The reaction I’ve seen has been somewhat mixed. Y’know, there obviously was sic some articles that came out, that sorta like, rightly pointed out that this was a bad thing that happened, and yet — I guess maybe you can speak to, because obviously you’re directly involved with it — the reaction from both individual users of Ma.gnolia, y’know, as well as, y’know, sorta like the larger media that’s like Wired and so on.

LH: So I think, um, the reaction has been actually mostly supportive and understanding.

CM: Yeah, I’ve seen a lot of that.

LH: I’d say ninety percent of what I’ve been getting and reading has been, y’know, not tearing down, not flaming, not griefing. And the negative reactions out there, I think a lot of them are valid. It’s… I made a huge mistake in terms of how I set up our system, and the — when people criticize that, they’re completely right. I have no problem with that. There have been some personal attacks, but… I think people get frustrated, rightly frustrated, and angry and sort of fall back into that mode, where they want someone to go after, and make it personal since they feel like they were personally let down.

CM: It’s also been interesting to see the characterizations of ‘Ma.gnolia and co.’, or ‘Ma.gnolia and team’, as though you’re this large operation, y’know, with international offices and things like that. I mean…

LH: I think that’s another lesson learned, which is, like, we always appeared bigger than we were. And it was me, and it’s basically been me. As of late, there was… for much of Ma.gnolia’s life, there was a small team; I think the largest we ever got was four. So we somehow projected this image of this, you know, big company with, you know, huge offices and cubicles and the whole works, and it was, you know… it’s really just, it’s really just basically me. And I don’t think… I mean, I think it was flattering that that’s the perception, but I think it was a mistake to not work harder to let people know exactly what we were and how big we were, in terms of how personal the service actually is.

CM: That says, I think that says a lot to a lot of the lessons coming out of social media lately; I think, y’know, around transparency and openness, which, obviously, Obama says a lot about, but there is some degree of truth there. Now…

LH: I don’t think it’s something we ever hid.

CM: Right.

LH: In fact, I was always surprised when… in terms of how large people thought they were. In fact, I was surprised at how much news coverage this whole event got, because Ma.gnolia is very small, even in terms of its user base, it’s very, very small. It’s just com… insignificant compared to any of the other real-web applications out there. But it somehow always projected this image of being this much bigger thing than it actually was.

CM: Yeah, I mean, even though it’s a small team, or just you, most of the time, I don’t think that that necessarily excuses what happened, but helps, maybe, to put in perspective, y’know, both from this hardware perspective, y’know, mostly you keeping it up and running, mostly you doing a lot of the work on these things that… I guess there are two things that can come out of this. One is that an individual can actually build a fairly, y’know, substantial community, relatively speaking, with the tools that exist today… that a lot of these tools are more accessible than maybe they once were. For example, you mentioned that the commodity-hosting thing sort of, y’know, that was the way that you could do it, which isn’t great; or you could do it yourself, and bear those possible risks and consequences. But it also says, I mean — this is, maybe, y’know, this is an opportunity to go back to where Ma.gnolia came from. I mean, I found Ma.gnolia a long time ago largely because I read the web standards book, by the ‘blue beanie guy’.

LH: Jeffrey Zeldman.

CM: Exactly. And he, of course, Happy Cog did the design of Ma.gnolia originally, and that’s how I originally found it. And Jason Santa Maria did the logo, and I was like, wow, this is a great-looking site, I really wanna use this, y’know, it looks kind of interesting. And yet I had no real insight into where it had come from. I mean, maybe you can talk about the germination of the site, the work that came before, that led to Ma.gnolia, and y’know… what maybe your goals were originally.

LH: Well, it’s been a sort of long and shifting road, but to go way way back, my background is in cultural anthropology, and I did my graduate work developing qualitative research tools. And sort of… I think I sort of revisit that every so often, and Ma.gnolia’s one of the revisitations of that work, and in a way it’s a tool that helps people gather disparate information, and thread it together in ways that make sense to them and their community. So, that’s sort of like the way-back origin of Ma.gnolia.

CM: Well what were your goals back then?

LH: So I think… I mean, it’s funny, I think my goals when I started Ma.gnolia were were to — at that point, you could make a lot more money on site advertising, so the idea was to build, was much more around a publishing model, and… sort of, as we launched the site, and as we watched people starting to use it, it was clear that that was not what this was going to be. And also was not necessarily true to my background and my work and who I was. And so, as… throughout the beta period, and the initial months, and the launch, we pretty quickly refocused the site on collaborative, community, developing-type tool, rather than just a publishing/advertising-type site. And in fact, ads were designed in the original site, and I never turned them on until… well, I think actually they were on briefly, but basically I left ads off the site until I added the ads-off upgrade. So that’s sort of the initial start. And going down that road definitely was the right thing of the site. I think it really found its identity, and really had a vision and message as a community site.

CM: So would you say that overall, you know, notwithstanding what happened recently, the site was a success?

LH: I think the site was a success in terms of what it brought to people’s lives, the community it developed… it definitely was like attracting like, in terms of like attracted people who cared about their… environment, I think, capital-e Environment, in terms of, like, not just the way the site looked and the way the site acted and the interaction, but also, like, the people around them, and the other people on the site. It attracted people who were thoughtful and caring, I think. So, yeah, I mean, in that sense it was a great success, in terms of, I was able to build something I loved for people I liked and respected. The site didn’t ever actually make money, was not a business or financial success.

CM: So from that perspective, you essentially were bankrolling the project, kind of maybe out of a hope at some point it might turn into some sort of business or something, but for the most part it sounds like it was a labor of love, which a large number of people eventually ended up sort of relying on and using on a fairly regular basis.

LH: Yes. This was definitely a labor of love. I was doing this because I loved to do it. And it was completely self-funded. I would have loved to, and I was working towards building a business model around it, with the add-ons, and I was working towards, y’know, bringing ads back into the site in a way that was more relevant. But some of, a lot of those plans never got executed on.

CM: Well, so let’s talk about that. I mean, there are a couple things that have changed in the last several years, largely, many more people are using social networks now, and there’s a lot more people online. The competitive marketplace for sharing bookmarks is probably heated up a bit, even though delicious is probably still the heavyweight, y’know, in the area, just because of Yahoo!’s involvement…

LH: I actually think, I mean, I think Ma.gnolia was a unique thing in terms of the community that organized around it. But I think that, in a way, I don’t think we could ever compete in the real world of link-sharing. I think the biggest link-sharing site right now is Facebook. The people are sharing their stuff: photos, links, any of the stuff in context of the communities they’re already hanging out in.

CM: Sure.

LH: So the destination of social bookmarking, I’m not sure where that’s going.

CM: So let’s talk about that, then. I mean, obviously, there’s sorta been a quiet lull after the storm, if you will, where I think, you know, you need an opportunity to maybe collect your thoughts about what happens next. But what are you sort of leaning towards right now? I mean, not all the data has been retrieved yet, or recovered yet; there are a number of tools that you’ve made available, which you probably could talk a little bit about, but in any case, whether the data is able to be retrieved, and people are able to get their bookmarks out of Ma.gnolia, is the question of, well, what happens in the future? Y’know, three, six, nine months from now, has Ma.gnolia recovered, has it come back? Because I think if you make that distinction between the data and the community, there’s something there. There’s a social fabric that was created that, though, ripped out of a context because the social objects went away, there’s still people who probably would like to continue connecting and sharing with one another. So…

LH: Yeah, and the community has asked for the tool back.

CM: Yeah, they want the tool.

LH: So that’ll be coming back, in a modified, in a different sort of way. It’ll be coming back in a proper hosting environment, for one thing.

CM: Where you’re not responsible for IT any more?

LH: Yeah. It is gonna go into a more reliable utility…

CM: Where Werner Vogels is responsible, the guy over at Amazon?

LH: Exactly. And with better backup safeguards in place. I think that’s my first priority in bringing it back, is… I mean, you could never guarantee anyone a hundred percent of anything, but I can get a lot closer than I was in the prior setup. So that’s sort of the biggest change that’s gonna happen, in terms of technically how the site is gonna change. In terms of how the community is gonna change, it’s gonna, it’s going to… it’s sort of, I think as like going back into private beta, that I’m not going to have it open registration, that the site is going to relaunch by invitation only, and then slowly build up from there. And definitely people who were part of Ma.gnolia 1, who were good community members there, will be invited back to join from the outset.

CM: Now do you think, I mean, that people can trust you again? Or do you think this is just gonna be something that you earn back over time?

LH: I think it’s gonna be something I earn back over time. I’m gonna completely disclose what our infrastructure is, when that’s settled on, and let people make the call based on that.

CM: Yeah, I think, y’know, it’s sort of raised a number of questions, I guess, in my mind, about… y’know, again, without that kind of, y’know, Apple menu, you see what’s behind these services. I use a lot of web services, and I had about 6300 bookmarks on Ma.gnolia, but I have similar sort of quantities of stuff — data capital, as I call it — strewn throughout the web on other services, for which I have no concept or idea of how they perform backups. So it’s been interesting for me to, y’know, witness some folks in the Get Satisfaction forums were coming and, y’know, making these claims about oh, this is preventable, and you could’ve done this or done that, and sort of, y’know, playing armchair IT professional and saying, well you could’ve done all these different things, but at the same time we don’t have a great deal of disclosure from other web services too. So it’s not just that Ma.gnolia was the only one doing this, it’s that there was an experience here that sort of sheds some light on these different IT practices, I guess, for better or worse. I imagine that there are a lot of other, for example, applications out there, y’know, that are built to serve the Twitter community, that are probably equally, if not much worse off, than Ma.gnolia from an infrastructure perspective. So it raises a broader question, I think, about, y’know, who we are entrusting with our data, and where we’re putting it, where we’re hosting it… And in some ways, making sure that there is a personal sort of connection or relationship there, I think it becomes more important over time. I mean, if you imagine these services as kind of like your bank, and you wanna entrust your bank, y’know, over time I think that individuals, now that you’ve had this experience, you’re never gonna repeat this problem, y’know, this situation. Other services may have to similarly have that kind of experience until we realize this is actually a big deal, and this is a long-term sort of, y’know, consideration to make. I mean, a lot of the work that, let’s say, I do with OpenID is around also helping OpenID providers understand and realize the gravity of their purpose. Y’know, it’s just like if your email went away, what would you do? For a lot of people, I think that would be very very bad. So there’s that, is that question, like, sure, people could keep countless backups of their own data on their own machines, and things like that; and only recently, though with tools like Time Machine has even personal backups become somewhat more accessible. So this is, I think, a question for many more people than just either Ma.gnolia or the Ma.gnolia community or you. Where? it’s a question of, how do we go about making smarter decisions about where we host our data? And just because we can keep everything, what is the real value? And I think it’s yet to be seen; I mean, you talk about sort of the qualitative… what was it, the research that you said?

LH: ? Collaborative qualitative research tools.

CM: Yeah, so you can imagine that this… I have this sort of abstract concept of this, since I don’t really have to get too much into the bits and bytes. I don’t really need to think about how hard it would be to do this the right way, but, y’know, 6300 bookmarks gives you kind of a fingerprint of the stuff that I’ve consumed over the last, y’know, three or four years. You could imagine using that as a filter for things that might interest me in the future. And so, on the one hand, just having your bookmarks some place, to me, is not all that interesting. I have backups, y’know, from ages ago, and I have stuff I did in college on hard drives somewhere. I’m never gonna look at that stuff again, but I have this sort of abstract through that, oh, some day I’ll break out the old, y’know, 180 gig hard drive, or actually at this point probably 180 megabyte hard drive; I’ll be like, oh, take a look at that! Y’know, like, I did that with Photoshop 3. But… there’s just so much data now that you almost need to be living much more in the moment, doing these things in real time, where a bookmark has a half-life of, y’know, twenty-four hours, if not less.

LH: Yeah. I mean, I think you’re right. I mean, the interesting value in data like bookmarks is probably more along the lines of what… I mean, there is stuff you wanna go back and find, but a lot of the value is probably in terms of using that to build other tools, like what Apple did with the iTunes Genius —

CM: That’s right.

LH: — where it’s like, they can look at your entire music collection —

CM: And your listening habits.

LH: — and your listening habits and stuff… you may not be listening to, right now, an album you got five years ago, but it can bring that back, or use that to find other songs, in terms of developing those Genius playlists.

CM: I mean, y’know, if you think about it from that perspective, Ma.gnolia has — or had, and may still have — a great opportunity to start doing that, where it could be, y’know, your daily list of links, recommended to you based on your previous history. And that’s something I haven’t seen done a great deal; it’s very hard to do, very hard to get right.

LH: It’s computationally intensive.

CM: That’s right. But nonetheless, you can imagine that moving forward, that would be a very valuable way of making use of, y’know, this type of service. So, well anyways, maybe you… any parting thoughts? Like, y’know, to, let’s say, Ma.gnolia users, y’know, who are sort of waiting for something. Y’know, the next thing, like…

LH: Um… I mean, I just can’t thank people enough for their support. It’s really… as difficult as this experience has been for me, I think my faith in humanity has been reaffirmed…

CM: That’s good.

LH: And really, I have everybody out there who was hurt by this experience to thank for that. And just, also, keep an eye out for the updates on Twitter and Get Satisfaction and the Ma.gnolia homepage for in terms? of bringing the community back, I’d say in the next month, month and a half.

CM: That’s, that’s exciting. I mean, I’m looking forward to it regardless. Y’know, whether or not my bookmarks are there or not is actually not what I’m most interested in. I think that having it there, it’s one of those things where, you don’t really miss it until it’s gone, right. So now that we’ve had that sense, you wanna fill that void, and I think it’s good to know that, y’know, Ma.gnolia will be, will grow again.

LH: It will.

CM: All right, well, appreciate it.

LH: Thank you.

Transcription of Citizen Garden episode 9, ‘Opening Preconditions’.

This one was informative for me because although I was vaguely aware of Ma.gnolia’s plans for their version two release, I didn’t know about the technical aspects — requiring OAuth, self-hosting, and open source code.

The discussion about using OpenID and related ‘open web’ technologies to automatically tell web services where things is quite appealing. This podcast episode is from late August, so maybe work has already begun on implementing such concepts.

But the most interesting part, to my mind, was the idea of using the fact that services tend to come in categories — bookmarks, social networks, and so on — we can export data from those services on a regular schedule and back it up for safekeeping. Then, if we decide to move services (e.g. MySpace to Facebook) or the service goes offline, we’ll have full access and control over our ‘social objects’.

If you think about it, this is really why the open web movement exists. The open web revolves around the idea that if I put information into the system then that data belongs to me. If I fill out a Facebook profile or use Twitter or make a bookmark on delicious, it’s true that somebody owns the system that allows those things — but what use would the system be if I didn’t use it?

So it’s about ownership. But ownership implies freedom — freedom to do whatever I want with my information, since I control who has it and what they can do with it.

This is something that companies offering hosted services dislike. Their business model is generally based on the idea that once you put your information into the system it’s stuck there. Investors see much more revenue potential in a captive audience, because the audience has no choice but to use whatever’s being offered, unimpressive as it may be — a non-technical example would be product placement.

And this is where the whole idea really becomes useful. An important part of the open web is data portability — the ability to easily transfer my information between services. Suppose I’m on Facebook, and I decide I want to try a different social network. With data portability, I’m able to tell the new site about my existing Facebook account and let it import all my friends.

This means that the power situation is changed. I’m no longer stuck using whatever site my friends have chosen. I’m no longer forced to trust that the site I’m on will continue to provide new features and a useful service. Instead, by using a service I’m actually endorsing it, because I’m not obligated to stay there.

It also entices the site’s owner to improve their service. When a site maintains its user base through feature offerings, they’re less able to simply make something good and then let it sit. Innovation is hard, but when it works everybody wins.

But despite all that, I set out to publish a transcript, not an essay. Here we go!

Participants

• Larry Halff
• Scott Kveton
• Chris Messina
• Will Norris

Transcript

LH: Hello!, and welcome to episode nine of Citizen Garden. I’m Larry Halff…

CM: I’m Chris Messina.

SK: I’m Scott Kveton.

WN: And I’m Will Norris.

LH: And we’re here today… I’m freshly back from Seattle after making a big announcement at Gnomedex that Ma.gnolia is being rewritten from the ground up as an open source, downloadable tool. Actually, it’s being broken up into several pieces. I think that’s the more important announcement; there’s a lot of open source publishing platforms out there… blogging tools, there’s even, y’know, I think, a handful of open source social bookmarking systems. A more important thing about what we’re doing is that we’re really trying to pave the way forward with the open web, and part of that is getting these decentralized and federated systems talking to each other. And open source happens to be one way to advance that cause.

CM: Let’s back up here a little bit, ‘cause I’d like to sort of understand better where this idea came from — why you would take this approach when, y’know, you’ve built up a pretty good audience on Ma.gnolia, you’ve got a good user base there, and you’ve done really well so far in supporting a number of these open protocols like OpenID and supporting microformats and OAuth and so forth. but how does open source or open sourcing the platform actually support the open web — and what actually motivated that decision?

LH: Well, I think the primary motivation was seeing that the social aspects of these publishing tools doesn’t really scale when there’s the big single point of failure. and Ma.gnolia has had downtime, Twitter has the infamous ‘fail whale’, Flickr gets massages — and all these things happen, and when it happens you sort of… you lose access to a pretty important piece the flow of your online life. And as they grow, the load that’s caused by exponentially putting out… the exponential effect of putting out all these social connections and publishing and keeping everyone up-to-date just doesn’t… just isn’t really gonna scale in the long term. And I think, y’know, what we’re seeing with Twitter now is nothing like what we’re gonna see with that kind of tool down the road. I mean, hardly anyone really uses Twitter…

SK: Yeah, like a million users maybe.

LH: So I think we really see an important next step is to finding out a way that these things can be pushed out to the edges, yet still have the social functionality of getting everyone talking to one another.

CM: Now one of the things I think, y’know is interesting… so, Scott is with us at Vidoop; he’s also the chair of the OpenID Foundation, and he’s, y’know, sort of one of the champions of decentralizing identity and things like that. I think it’s sort of interesting to think about how OpenID in some ways creates the preconditions for Ma.gnolia going open source, by allowing people essentially to have sort of a cargo horse on which they stack a bunch of things — their photos, their bookmarks, and so forth. Maybe you can — Scott — talk to us a little bit about how you see OpenID maybe as that beginning point that… allows for this type of decentralization that Larry’s talking about, where there are much fewer single points of failures, or at least these concentrations that maybe are creating these pressures on the network.

SK: Okay! Uh, yeah, absolutely! So there’s, y’know, we’re three years into OpenID now, and it was really funny at OSCON, actually — and I think Chris you were on that panel with me, weren’t you?

CM: Possibly, yeah.

SK: Leah Culver got up…

CM: Oh, right.

SK: … started just dropping f-bombs about how she just thought OpenID was dumb, she’d never use it, da da da da. And y’know, in reality, it’s… it is a URL-based system, and that’s just… users have not, y’know, grokked that. But I think what we’re starting to realize is the value is proving a potential service endpoint, which means nothing to users.

CM: [Nor necessarily?] should it…

SK: Right, absolutely.

CM: For now.

SK: … but for developers it’s extremely important. So, y’know, some of the challenges that we’ve been facing in the OpenID community are around security and usability. If we can make it easier for people to identify themselves — whether with an email address or identity in the browser — then they can get in, prove that they are some end point without having to know what that endpoint is, and then start to put their data there. and then we start that… that lays the groundwork for things like y’know, lower-case data portability and the ability to, y’know, have more control over who provides your… or manages your data.

CM: Yeah, I’ve been thinking about these things lately, and it… it’s interesting to reflect on what assumptions we bring to these problems, especially around sort of, as you talked about, the developer part of the equation, where you’re starting to think about and understanding well, if a developer starts to tackle a problem, trying to build a new service on the social web, what assumptions can they make today that they couldn’t make before? And previously maybe you made an assumption that people would have an email address before; okay, great, then you could send them a password or a token, and they can prove received that token — that’s a way of confirming… that’s a durable identifier, but you can’t really attach services to it. You can’t actually look up that email address and ask it, y’know, ‘well where is this person’s photo store’, y’know, ‘where is this…’ — there’s no directory for that. Using URLs as identifiers sort of helps to at least make that situation a little bit better. So I’m actually interested in hearing from Will, ‘cause Will and I work on the DiSo project together. How do you think the ability to use unique URLs to identify people — which then have services offered at the end of them — changes what you can sort of take for granted? What are the building blocks now, that when you approach building a new application, you’re like oh, well they probably have identity, they probably have, y’know, some service that they’re gonna use, we’re just gonna throw it all together and make it happen?

WN: Yeah, right. I mean, we’ve been trying to address those exact problems of ‘how can we build that infrastructure so that you can take advantage or take it for granted?’. And like you said, y’know, you can’t attach these services to email addresses. Recently, I co-authored a spec — and we’re now using this with a lot of our stuff — called ‘EAUT’, which is e.a.u.t. It’s Email Address to URL Transformation. It basically allows… it’s a really standard way of taking that email address and talking to whichever email provider it is that provides that, and say, y’know, ‘how can I turn this into a URL that I can then go and try to find these kinds of services on that URL?’. So y’know, Yahoo! can host their own thing, Google and all this, and then we also have this fallback service. But the idea is that, so, we can use an identifier that users are familiar with, and comfortable with, and use everyday, and they love — it’s their email address — but still be able to get these additional kinds of things. So, y’know, when I go in to whatever it is, I can give my email address, it can be converted into an OpenID; this application can then go and look at my OpenID and say ‘hey, y’know, we need to publish a bookmark, where should I do that?’ And, y’know, I could be doing that on Ma.gnolia proper, or if, y’know, I have my own Ma.gnolia instance running, with the new open source version, or if I’ve, y’know, maybe… maybe Vidoop, y’know, we set up our own, for our employees, I can say y’know what, that’s where I do mine. And so all I have to do is present my identifier, and, y’know, magic happens, basically, and this consumer can push that bookmark out to wherever it is that that I store them.

LH: Right. I think it’s important, it’s like that’s another precondition we haven’t explicitly mentioned. It’s like, we have the… we know identity exists, and we also, with the unsung but super-important spec XRDS Simple is really key to making that happen, because at least for programmers, it’s like, they can go to the end of your OpenID, and they can say ‘well, this is someone’s identity URL’, and they can get a whole bunch of links off of that. But they’re really meaningless until we have a way of saying ‘what do each of these links do?’, and so I think, I just… point that out as a key piece to that, which isn’t just having the thing, but also having the mechanism to say ‘what can we do with these different things?’.

CM: To put that another way, it’s kind of like when you go to someone’s blog and they have a sidebar that lists all their other profiles. What we’re trying to do, I think — and this is sort of what this discovery protocol is all about — is taking that list of URLs and making them make sense to computers, essentially. So that when it sees a Twitter icon, y’know, more or less, it’s like ‘oh, that’s a status update service’, or ‘that’s a microblogging service, therefore I can post messages to it, if I’m authorized to do so’. Very similarly, if you see a little Flickr icon, or if you see a YouTube icon, those are different services that someone might use that actually have APIs that you can talk to. So if you can advertise those URLs through this discovery/specifications at the end of an OpenID identifier, that’s where the magic starts to happen. So, what I was actually interested in hearing you two guys talk about a little bit — and recently, y’know, you’re wearing the WordCamp shirt, and you were at WordCamp, will, and you talked about OAuth for WordPress. and this is, I think, very interesting, because we’re at a point where WordPress currently does not support OAuth. Most of its transactions are done with the standard username and password, which means that if you wanna, let’s say, blog from your iGoogle home page, you’re gonna have to enter in your WordPress username and password into iGoogle. Well that’s great, except when you start doing this across the web and so on, and that’s the password anti-pattern. now on the other side, we have Ma.gnolia, which already supports OAuth in the platform. And I’m interested sort of in hearing you guys talk a little bit about the pros and cons and the challenges of retrofitting OAuth and authorization-based permissioning into a platform like WordPress, whereas Ma.gnolia already has that — Ma.gnolia open is gonna come out supporting that from the get-go — what does that mean for people building on these different platforms? How does that actually improve the situation?

LH: I think it improves the situation that enables a lot more seamless experience for the end user that… I think combining OpenID with discovery with something like OAuth is, y’know… this is a whole lot of hot air, so I’m ashamed of the words that are about to leave my mouth, but the browser of the future…

CM: Uh-oh.

LH: There, I did it.

?: He did it! He did it!

LH: … will be like…

SK: Tshirt’s already been ordered.

LH: The browser of the future…

CM: dot com!

SK: That’s right. Do we have that yet?

LH: … when I go to save a bookmark, instead of it saving in by browser — it will have known my OpenID already, because when I launch my browser and it’s first setup, it says ‘what is your OpenID?’ — it will have verified that, it will have discovered my online bookmarking service, and it will know where that lives, and as part of that process it will have authorized access to my bookmarking service account though OAuth, and I will have said ‘yes, this browser allowed to post things to my bookmarking service’. And so ‘save bookmark’, it will be seamlessly integrated to me. And that really is the end-user benefit for all this, despite all the horrible, geeky, completely incomprehensible nerdiness and the ongoing usability issues with OpenID, which Ma.gnolia open has cracked open again.

CM: Yes.

LH: If you wanna participate in a great little thread about OpenID usability, go to the magnolia-2-discuss group on Google.

SK: Yeah, y’know, if there’s one thing I’ve learned over the last three years of the OpenID stuff, is it doesn’t matter how open it is, if it’s not usable, it’s broken. And so that means, y’know, I think usability has to come first, and I think we have to break some things around the openness of it to get it right first. And we’re seeing, y’know, Facebook — we were talking about this today. Y’know, what Facebook is doing sort of, y’know, in the eyes of a lot of folks who are very open-centric — which doesn’t really make sense, but anyway — they see that as awful, because they…

CM: Whoa, wait; be more specific about what Facebook’s doing.

SK: Well, they’re embedding an <iframe>, effectively, on other sites — and this is effectively what Google is doing as well.

CM: That’s right, that’s right.

SK: And, y’know, that’s all well and good for the sites themselves — they don’t actually get the access to the user information — but, they can get more people and pageviews, which could be really important to them. but from an open perspective, it’s not that open, to be able to do that. And… what’s the other thing I was gonna say? God, this coffee really does — oh my god! sorry.

LH: These guys bought you Blue Bbottle before?

WN? Sponsored by Blue Bottle…

LH: I am writing without Blue Bottle here.

CM: It’s a plug for the [something]. Oh my god.

SK: It’s good stuff.

LH: But, so I mean… so I think we were getting… we were also talking about the whole convergence to this stuff, and what’s, like, what’s… the work you’re doing with WordPress, and like, where’s… do you see that headed in a similar direction? Is WordPress thinking about this?

WN: In a similar direction as..?

LH: As sort of, like, end-to-end integration…

CM: Well the interesting thing is that Weave — which is a Mozilla Labs project — is kind of in that direction, where it actually does kind of what you described, Larry, in allowing you to kind of sign in with some accounts, through OAuth actually authorize the browser to both publish your bookmarks and download ones that you’ve already saved — very much like a MobileMe for the rest of us, in a sense, for those of us who are not gonna pay Apple or whatever to do so. And all of… the entire sort of service stack of MobileMe could be more or less built on open technologies. It’s interesting, though, to think about what it would mean for someone like WordPress, or even someone like Drupals of the world and so on to really embrace some of these technologies, and to look at the opportunity that the browser, y’know, deep browser integration and web service access and offline storage, to some degree, would offer. And so I guess the question for Will is sort of around, y’know, what would OAuth mean to the WordPress platform? How would that accelerate the development of things, how might that make WordPress a different type of integral platform for publishing all sorts of different services on the web, perhaps?

WN: Well I guess, I mean… the most immediate use case, I think, that we’re gonna see with getting OAuth into WordPress is just allowing whatever service it is to publish to WordPress without needing the user’s credentials. So, this could be something like the WordPress iPhone app, this could be MarsEdit, it could be… Flickr already has a way where you can push your photos directly from Flickr into your WordPress blog…

CM: Well Ma.gnolia 2 actually offers publishing your bookmarks to your blog.

WN: Oh, does it?

CM: But right now, I believe, you have to take… it’s the password anti-pattern.

LH: Yeah. It’s the password… nobody… yeah. The major blogging platforms except for Blogger — which we don’t support because they don’t use the MetaWeblog API — use OAuth authentication. And I mean, that’s great for Google [something].

CM: And Movable Type is going to be supporting OAuth in the next release, I believe.

SK: It’s already out.

LH: It’s already out.

SK: The libraries are there, they’re in the core release… four, two, whatever it is.

CM: But still, we’re still at a point where we need that deeper integration, but…

SK: Well, yeah, and just kinda playing off that a little more is that a lot of people that I’ve seen are using WordPress as kinda more of a persistent storage of their social objects. Y’know, with Ma.gnolia, y’know, you have… there’s things going on with Ma.gnolia, but, y’know, Ma.gnolia might go away tomorrow, or I might wanna move to some other platform, so I wanna have a copy within my own control. So, y’know, I do a nightly pull or push or whatever to my blog. People do that with their Twitters — er, their tweets, y’know, they’ll do a day’s worth of tweets as a blog post.

?: My tweets are very important.

SK: Well yeah, absolutely, that’s how people feel. And whatever that…

LH: Live coverage of the Olympics…

?: [something] my addiction.

SK: So yeah, I mean… just simply using WordPress as kind of a persistent storage of these objects that are within the individual user’s control. And in order to make all that stuff happen in a secure way, yeah, absolutely, you’re gonna need a secure mechanism for pushing that in, and that’s gonna be OAuth at some point, once we get that built.

CM: And I think… well I dunno, we probably should wrap up pretty soon…

LH: I think we’re heading towards that.

CM: Yeah, well… so I guess actually we could close on some final thoughts, since this has sorta been a whirlwind discussion… and there’s much more, obviously, we could talk about. What it sounds like you’re talking about — I really like the way you framed it in terms of kind of your store for social objects, your generic store, is that increasingly we’re gonna have specific tools that do a good job at storing different types of social objects and providing metadata around those objects. So we’ll have Ma.gnolia, the Ma.gnolia bookmarks is like the WordPress of bookmarks in a sense, so you might use your self-hosted WordPress — I’m sorry, Ma.gnolia install — to host bookmarking-type things, which have certain screenshots of the webpages, maybe some tags, so on and so forth…

LH: And have access to all that’s going on out there.

CM: Right. And maybe you’ll also be able to push those bookmarks out, and also pull things in via that type of channel, because again, that channel is designed specifically for those types of objects. Then you have your Flickr, which might be a better photosharing application, or maybe you wanna use Facebook to view photos. I mean, who knows. Whatever the case is, moving these objects around into different web applications seems to be where this is gonna go, and being able to push the data around fairly [in a] fairly straightforward way using OAuth to control sort of who has access to read/write, that’s important, and then coming up with the standard protocols so that each endpoint kind of understands what kind of data is being pushed around is also a matter of import, I think, as well. So it’s really interesting to think about how we can actually move to real cloud computing using these types of protocols. So that was a longer closing from me, so what do you think?

SK: Yeah, I think… I think as we move closer and closer to having… putting users in control of their data — and I actually really like that term ‘social objects’ — because to me social networking actually isn’t something you do on the internet, it’s just a feature? And especially when you apply it to things like bookmarking services or photosharing sites, I wanna be able to bring my social network along with me. It should just be a foregone conclusion. And so to me, the work that we’ve all been doing has been headed in the direction. And, y’know… good stuff.

WN: Yeah, me too. Plus one. Plus one!

SK: Plus one for me.

LH: I think for me, it’s like, what we’re gonna be seeing next, since — is that mandatory, do you have to close with ‘what’s next’?

CM: No.

SK: Absolutely!

LH: But I’m gonna do it. What we’re gonna be seeing next is — and Ma.gnolia isn’t the first — but we’re gonnna be seeing these kind[s] of services becoming more decentralized, which means… which means creating another problem. But the ‘more decentralized’ means more reliability, more control, more adaptability to individuals’ needs. But that removes a lot of social functionality, removes a lot of community, removes a lot of interaction. So we’re gonna be seeing that problem solved. We already know how to decentralize; we do it with blogs. It’s there. But we’re gonna be seeing the federation problem being solved over the next few months. And we’re gonna see how we can bring those together in more of a ‘small pieces’ type solution [something] social network.

SK: We’re gonna solve the problem in six months?

LH: Yeah. Yeah.

SK: All right, let’s go.

CM: We’ll be there.

LH: High five.

?: Yeah!

CM: So just one more plug for… it’s Ma.gnolia.org is where you’re gonna find this stuff, and it’s ma-dot-gnolia-dot-org, that’s where you can find out more about the announcement, the m2 — as it’s being called — charter, and…

LH: And the Google group.

CM: Yeah.

LH: Come and join and contribute to the discussion.

CM: There you go.