“None Dare Call It Conspiracy”

Note: What follows is a transcript of Scott Anderson’s article for this month’s edition of GQ regarding the Russian apartment bombings in 01999. This article has been carefully surpressed:

Jerry Birenz ordered that the article could not be posted to the magazine’s Web site. No copies of the American edition of the magazine could be sent to Russia or shown in any country to Russian government officials, journalists or advertisers. Additionally, the piece could not be published in other Condé Nast magazines abroad, nor publicized in any way.

I was inspired after seeing that Gawker solicited a Russian translation; I’ve worked from the scans they provided.

This transcript is also available at my GitHub account.


Ten years ago this month, Russia was rocked by a series of mysterious apartment bombings that left hundreds dead. It was by riding the ensuing wave of fear and terror that a then largely unknown Vladimir Putin rose to become the most powerful man in the country. But there were questions about the nature of those bombings — and disturbing evidence that the perpetrators might actually have been working for the Russian government. In the years since then, the people who had been questioning the official version of events began one by one to go silent or even turn up dead. Except one man. Scott Anderson finds him.


The first building to be hit was the barracks in Buynaksk housing Russian soldiers and their families. It was a nondescript five-story building perched on the outskirts of town, and when the enormous truck bomb went off late on the night of September 4, 01999, the floors pancaked onto each other until the building was reduced to a pile of burning rubble. In that rubble were the bodies of sixty-four people — men, women, and children.


In the predawn hours of last September 13, I left my hotel in central Moscow and made for a working-class neighborhood on the city’s southern outskirts.

It had been twelve years since I’d been in the Russian capital. Everywhere, new glass-and-steel buildings had gone up, the skyline was studded with construction cranes, and even at 4 A.M., the garish casinos around Pushkin Square were going full tilt and Tverskaya Street was clogged with late-model SUVs and BMW sedans. The drive was a jarring glimpse at the colossal transformation that Russia, its economy turbocharged by petrodollars, had undergone in the nine years since Vladimir Putin came to power.

But my journey that morning was to a place in “old” Moscow, to a small park where a drab nine-story apartment building known as 6/3 Kashirskoye Highway had once stood. At 5:03 on the morning of September 13, 01999 — exactly nine years prior to my visit — 6/3 Kashirskoye had been blasted apart by a bomb secreted in its basement; 121 of its residents had died while they slept. That explosion, coming nine days after the one in Buynaksk, was the third of what would be four apartment-building bombings in Russia over a twelve-day span that September, leaving some 300 citizens dead and the nation in panic; it was among the deadliest series of terrorist attacks in the world until September 11. Blaming the bombings on terrorists from Chechnya, Russia’s newly appointed prime minister, Vladimir Putin, ordered a scorched-earth offensive into the breakaway republic. On the success of that offensive, the previously unknown Putin became a national hero and swiftly assumed complete control of the Russian state. It is a control he continues to exert today.

Where 6/3 Kashirskoye had stood there was now an orderly grid of well-tended flower beds. These surrounded a stone monument engraved with the names of the dead and topped by a Russian Orthodox cross. For the bombing’s ninth anniversary, three or four local journalists had shown up, discreetly watched over by a couple of policemen in a nearby squad car, but there really wasn’t much for anyone to do. Shortly after 5 A.M., a cluster of perhaps two dozen people — most of them young, relatives of the dead, presumably — trooped up to place candles and red carnations at the foot of the monument, but they retreated as quickly as they had appeared. The only other visitors that morning were two elderly men who had witnessed the bombing and who dutifully related for the television cameras how terrible it had been, such a shock.

I saw that one of the old men became quite emotional as he stood before the monument, repeatedly brushing at his cheeks to wipe away tears. Several times he turned and walked purposefully away, as if willing himself to leave, but he never got very far. He would linger by the trees at the edge of the park and then inevitably make a slow return to the shrine. Finally, I approached him.

I lived very close to here, he said, and I was awoken by the sound, I came rushing over and… He was a big man, a former sailor, and he waved his hands helplessly over the flower beds. Nothing. Nothing. They pulled a young boy and his dog out. That was all. Everyone else was already dead.

But as it turned out, the old man had a more personal connection to the tragedy. His daughter, son-in-law, and grandson had lived at 6/3 Kashirskoye, and they had all perished that morning, too. Leading me up to the monument, he pointed out their names in the stone, and desperately brushed at his eyes again. Then he angrily whispered: They say it was the Chechens who did this, but that is a lie. It was Putin’s people. Everyone knows that. No one wants to talk about it, but everyone knows that.

It is a riddle that lies at the very heart of the modern Russian state, one that remains unsolved to this day. In the awful events of September 01999, did Russia find its avenging angel in Vladimir Putin, the proverbial man of action who crushed his nation’s attackers and led his people out of a time of crisis? Or was that crisis actually manufactured to benefit Putin, a scheme by Russia’s secret police to bring one of their own to power? What makes this question important is that absent the bombings of September 01999 and all that transpired as a result, it is hard to conceive of any scenario whereby Putin would hold the position he enjoys today: a player on the global stage, a ruler of one of the most powerful nations on earth.

It is peculiar, then, how few people outside russia seem to have wanted that question answered. Several intelligence agencies are believed to have conducted investigations into the apartment bombings, but none have released their findings. Very few American lawmakers have shown an interest in the bombings. In 02003, John McCain declared in Congress that there remain credible allegations that Russia’s FSB [Federal Security Service] had a hand in carrying out these attacks. But otherwise, neither the United States government nor the American media have ever shown much inclination to explore the matter.

This apparent disinterest now extends into Russia as well. Immediately after the bombings, a broad spectrum of Russian society publicly cast doubt on the government’s version of events. Those voices have now gone silent one by one. In recent years, a number of journalists who investigated the incidents have been murdered — or have died under suspicious circumstances — as have two members of Parliament who sat on a commission of inquiry. In the meantime, it seems that most everyone whose account of the attacks ran counter to the government’s version now either refuses to speak, has recanted his earlier statements, or is dead.

During my time in Russia this past September, I approached a number of individuals — journalists, lawyers, human-rights investigators — who had been involved in the search for answers. Many declined to speak with me altogether. Others begrudgingly did so but largely confined their statements to a recitation of the known inconsistencies in the case; if pressed for an opinion, they allowed only that the matter remained “controversial.” even the old man in Kashirskoye park ultimately underscored the air of unease that hovers over the topic. After readily agreeing to a second meeting, at which he promised to introduce me to other victims’ families who doubted the government’s account, he had a change of heart.

I can’t do it, he said when he called me back a few days later. I spoke to my wife and my boss, and they both said that if I meet with you, I will be finished.

I was curious what he meant by “finished,” but the old sailor hung up before I could ask.

No doubt part of this reticence stemmed from recalling the fate of the man who made proving the conspiracy behind the bombings a personal crusade: Alexander Litvinenko. From his London exile, the rogue former KGB officer had waged a relentless media campaign against the Putin regime, accusing it of all manner of crimes and corruption — and most especially of having orchestrated the apartment-building attacks.

In November 02006, in a case that riveted the world’s attention, Litvinenko was slipped a lethal dose of radioactive polonium, apparently during a meeting with two former Russian intelligence agents in a London hotel bar. Before the poison killed LItvinenko — it took an agonizing twenty-three days — he signed a statement placing the blame for his murder squarely at Putin’s feet.

But Litvinenko had not worked alone on the apartment-bombing case. Several years before his murder, he had enlisted another ex-KGB agent in his search for answers, a former criminal investigator named Mikhail Trepashkin. The two men had a rather complicated personal history — in fact, back in the ’90s, one allegedly had been dispatched to assassinate the other — but it had actually been Trepashkin, working on the ground in Russia, who had uncovered many of the disturbing facts in the case.

Trepashkin had also run afoul of the authorities. In 02003 he had been shipped off to a squalid prison camp in the Ural Mountains for four years. By the time of my visit to Moscow last year, however, he was out on the streets again.

Through an intermediary, I learned Trepashkin had two young daughters, as well as a wife who desperately wanted him to stay out of politics; combining these factors with his recent prison stint and the murder of his former colleague, it seemed likely that my approach to him would go as badly as had my conversations with other former dissenters.

Oh, he’ll talk, the intermediary assured me. The only way they’ll stop Trepashkin is by killing him.


On September 9, five days after the blast in Buynaksk, the bombers struck Moscow. This time it was an eight-story apartment building on Guryanova Street, in a working-class neighborhood in the city’s southeast. Rather than a truck bomb, the device had been stashed on the building’s ground floor, but the result was virtually identical; the explosion brought down all eight floors and killed ninety-four residents as they slept.

It was with Guryanova Street that the general alarm first went out. Within hours a number of Russian-government officials strongly suggested that terrorists from Chechnya were responsible, and the nation was sent into a state of high alert. As thousands of police fanned out to question — and in several hundred cases, to arrest — anyone resembling a Chechen, residents of apartment buildings throughout Russia organized themselves into neighborhood-watch patrols. Calls for retaliation rose from all political quarters.

At Trepashkin’s request, our first meeting took place at a crowded coffee shop in central Moscow. One of his aides showed up first, and then twenty minutes later Trepashkin arrived in the company of his bodyguard of sorts, a muscular young man with a crewcut and an opaque stare.

Trepashkin, while short, was powerfully built — a testament to his lifelong practice of a variety of martial arts — and still very handsome at 51. His most arresting feature, though, was a perpetual amused grin. It gave him an aura of instant likability, friendliness, although I could imagine that anyone who sat across an interrogation table from him back in his KGB days might have found it unnerving.

For a few minutes, we chatted about everyday things — the unusually cold weather in Moscow just then, the changes I’d noticed since my last visit — and I sensed Trepashkin was trying to figure me out, deciding how much to say.

Then he began to tell me about his career at the KGB. He’d spent most of his years as a criminal investigator who specialized in antiques smuggling. He was, in those days, an absolute loyalist to the Soviet state — and most especially the KGB. Trepashkin was such a dedicated Soviet that he even supported a group that attempted to thwart the ascent of Boris Yeltsin in favor of preserving the Soviet system.

I could see that this was going to be the end of the Soviet Union, Trepashkin explained in the coffee shop. But even more than that, what would happen to the KGB, to all of us who had made it our lives? I saw only disaster coming.

And that disaster came. With the disintegration of the Soviet Union, Russia plunged into economic and social chaos. One particularly destructive aspect of that chaos stemmed from the vast legions of Russian KGB officers who suddenly entered the private sector. Some went into business for themselves or joined on with the mafiyas they had once been detailed to combat. Still others signed on as “advisers” or muscle for the new oligarchs or the old Communist Party bosses who were frantically grabbing up anything of value in Russia, even as they paid obeisance to the “democratic reforms” of President Boris Yeltsin.

Of all this, Trepashkin had an intimate view. Kept on with the FSB, the Russian successor the the KGB, the investigator found it increasingly difficult to differentiate criminality from governmental policy.

In case after case, he said, there was this blending. You would find mafiyas working with terrorist groups, but then the trail would lead to a business group or maybe to a state ministry. So then, was this still a criminal case, or some kind of officially sanctioned black operation? And just what did ‘officially sanctioned’ actually mean anymore, because who was really in charge?

Finally, in the summer of 01995, Mikhail Trepashkin began work on a case that would change him forever, one that placed him on a collision course with the seniormost commanders of the FSB and, Trepashkin says, would lead at least one of them to plot his assassination. As with so many other incidents that exposed the malevolent rot in post-Soviet Russia, this one centered on events in the breakaway southern republic of Chechnya.

By December 01995, rebels fighting for the independence of Chechnya had fought the Russian army to a bloody and humiliating stalemate after a full year of war. The Chechens’ success was not as simple as mere force of arms, however. Even during the Soviet era, Chechen mafiyas had controlled much of the Russian criminal underworld, so when Russian society itself became criminalized it played beautifully to the Chechen rebels’ advantage. For their steady supply of sophisticated weapons with which to fight the Russian army, the rebels often had only to turn to corrupt Russian army officers who had access to such weaponry, with the funds for such “purchases” supplied by the Chechen crime syndicates operating throughout the nation.

Just how high up did this cozy arrangement go? Mikhail Trepashkin got his answer on the night of December 1, when a team of FSB officers stormed a Moscow branch of Bank Soldi with guns drawn.

The raid that night was the culmination of an elaborate sting operation, one that Trepashkin had helped supervise, designed to finally bring down a notorious bank-extortion team linked to a Chechen rebel leader named Salman Raduyev> It was a huge success: Caught up in the Soldi dragnet were some two dozen conspirators, including two FSB officers and a Russian-military general.

But inside the bank, the FSB men found something else. To ensure they weren’t walking into a trap, the conspirators had planted electronic bugs throughout the building, and those were linked to an eavesdropping van parked outside. While their precautions obviously needed some fine-tuning, it begged the question of how the gang got their hands on bugging equipment.

All these sorts of devices have serial numbers, Trepashkin explained in the Moscow coffee shop, and so we traced the numbers back. We discovered that it had all come from either the FSB or the Ministry of Defense.

The implication of this was staggering, for access to such equipment was severely restricted. It suggested that high-ranking security and military officers had colluded not only with a criminal gang but with one whose express purpose was to raise funds for a war against Russia. By the standards of any country, that wasn’t just corruption, it was treason.

Yet no sooner had Trepashkin started down that investigative trail than he was removed from the Bank Soldi case by Nikolai Patrushev, the head of the FSB’s internal-security department. What’s more, he says, no charges were brought against any of the Russian officers implicated, and nearly all of those caught in the initial dragnet were soon quietly released. Instead, Patrushev ordered an investigation of Trepashkin. That investigation lasted nearly two years, at the end of which Trepashkin had reached his personal breaking point. In May 01997, he wrote an open letter to President Yeltsin detailing his involvement in the case and charging much of the senior FSB leadership with a host of crimes, including forming alliances with mafiyas and even recruiting their members into FSB ranks.

I thought that if the president knew what was happening, Trepashkin said, then he would do something about it. This was a mistake on my part.

Indeed. Boris Yeltsin, it turned out, was fabulously corrupt himself, and the letter alerted the FSB that they now had a serious malcontent on their hands. The very next month, Trepashkin resigned from the FSB, burnt out, he says, by the harassment he’d been subjected to. But that didn’t mean Trepashkin was going to go quietly into the night. That summer he brought a lawsuit against the FSB leadership and began filing complaints that extended all the way to the FSB director himself. It was as if, even at this late date, the investigator imagined that the honor of the Kontora (Bureau) could still be redeemed, that some as yet invisible reformer might step forward. Instead, his persistence apparently convinced some senior FSB officials that it was time for a permanent solution to their Trepashkin problem. One of the first people they turned to was Alexander Litvinenko.

On paper, Litvinenko looked just the man for the job. Having just returned to Moscow from a stint on the brutal Chechen battlefield as a counterterrorism operative, he had been transferred into a new and highly secretive of the FSB called the Office for the Analysis of Criminal Organizations, or URPO. While Litvinenko didn’t know it at the time, it seemed the URPO had been formed to serve as a death squad. As reported in the book Death of a Dissident, by Alex Goldfarb and Litvinenko’s widow, Marina, Litvinenko learned of this when he was summoned by the URPO commander in October 01997. There is this guy, Mikhail Trepashkin, the commander allegedly told Litvinenko. He is your new object. Go get his file and make yourself familiar with it.

When he did, Litvinenko learned of the criminal investigator’s involvement with the Bank Solid case, as well as his lawsuit against the FSB leadership; it left him puzzled as to just what he was supposed to do with Trepashkin.

Well, it’s a delicate situation, Litvinenko quoted his commander as saying. You know, he is taking the director to court and giving interviews. We should shut him up, director’s personal request.

Shortly after, Litvinenko claimed his target list expanded to include Boris Berezovsky, an oligarch and Kremlin insider whom apparently someone powerful now wanted dead. Litvinenko stalled for a time, making continual excuses for his inability to carry out the assassination orders.

According to Trepashkin, at least two attempts were made on his life during this period: a failed ambush on a deserted stretch of Moscow highway, and a rooftop sniper who couldn’t get off a clean shot. On other occasions, he says, he was tipped off by friends still in the Kontora.

In November, the alleged FSB plot against Trepashkin and Berezovsky was exposed in dramatic fashion when Litvinenko and four of his URPO colleagues appeared at a Moscow news conference to detail the kill orders they’d been given. Also in attendance was Mikhail Trepashkin.

And there, somewhat anticlimactically, the matter seemed to end. Litvinenko, the ringleader of the dissident officers, was summarily dismissed but otherwise suffered no immediate retribution. As for Trepashkin, after improbably winning his lawsuit against the FSB, he married for a second time and settled into his new job with the Russian tax police, determined, he says, to quietly serve out his term until he was eligible for retirement.

But then, in September 01999, the apartment-building bombings would shake Russia’s political foundations to their core. Those attacks would also propel Trepashkin and Litvinenko back into the shadow world, this time with a common purpose.


Amid the near hysteria that gripped Moscow after the Guryanova Street bombing, early on the morning of September 13, 01999, authorities were called to check on reports of suspicious activity at an apartment building on the city‘s southern outskirts. Finding nothing untoward, security personnel completed their search of 6/3 Kashirskoye at about 2 A.M. and left. At 5:03 A.M., the nine-story building was collapsed by a massive bomb, leaving 121 civilians dead.

Three days later, the target was an apartment building in Volgodonsk, a city south of Moscow. This time it was a truck bomb, and it left another seventeen dead.


In the Moscow coffee shop, Trepashkin grew uncharacteristically somber, staring into the distance for a long moment.

It just seemed incredible, he said finally. That was my first thought. The country is in an uproar, vigilantes are stopping strangers on the streets, there are police roadblocks everywhere. So how is it possible that these bombers are moving about so freely, that they have all this time to set up and carry out these sophisticated bombings? It seemed impossible.

Another aspect that Trepashkin had a problem with was the question of motive.

Usually, this is quite easy to find, he explained, it is money or hatred or jealousy, but for these bombings, what was the Chechens’ motive? Very few people thought about this.

On one level, this was perhaps understandable. Antipathy for Chechens is deeply ingrained into Russian society, and it had grown much worse during their secessionist war in the ’90s. Unspeakable atrocities were committed by both sides in that conflict, and the Chechen rebels had shown no compunction against taking their fight into Russia proper or targeting civilians. Except that war had ended in 01997, with Boris Yeltsin signing a peace agreement recognizing Chechnya’s autonomy.

So why? Trepashkin continued. Why would the Chechens want to provoke the Russian government when they already had everything they had fought for?

And there was something else that gave the former criminal investigator pause: the composition of the new Russian government.

In early August 01999, just weeks before the first bombing on Buynaksk, President Yeltsin had appointed his third prime minister in less than three months. He was a slight, humorless man, virtually unknown to the Russian public, named Vladimir Putin.

The chief reason he was so little known was that, until a few years earlier, Putin had been just one more midlevel KGB/FSB officer toiling away in obscurity. In 01996, Putin was given a position in the presidential-property-management department, a crucial office in the Yeltsin patronage machine that gave Putin leverage to grant or withhold favors to Kremlin insiders. He apparently put his time there to good use; over the next three years, Putin was promoted to deputy chief of the presidential staff, then to director of the FSB, and now to prime minister.

But though Putin was still obscure to the general public in September 01999, Mikhail Trepashkin already had a pretty good sense of the man. Putin had been the FSB director at the time the URPO scandal went public and had personally dismissed Alexander Litvinenko for provoking it. I fired Litvinenko, he had told a reporter, because FSB officers shouldn’t hold press conferences… and they shouldn’t make internal scandals public.

But equally alarming to Trepashkin was who had been chosen to be Putin’s successor as FSB director, Nikolai Patrushev. As head of the FSB internal-security department, it was Patrushev who had removed Trepashkin from the Bank Soldi case and who was now among those government officials most vehemently claiming a Chechen connection to the apartment-building bombings.

So what you saw was this dynamic building, Trepashkin said, and it was the government promoting it. ‘The Chechens are behind this, so now we must take care of the Chechens’.

But then something very strange happened. It happened in the sleepy provincial city of Ryazan, some 120 miles southeast of Moscow.

Amid the state of hypervigilance that had seized the nation, several residents of 14/16 Novosyolov Street in Ryazan took notice when a white Zhiguli sedan pulled up to park beside their apartment building on the evening of September 22. They became downright panicked when they observed two men removing several large sacks from the car’s trunk and carrying them into the basement before speeding away. Residents called the police.

Discovered in the basement were three 110-pound white sacks wired to a detonator and explosive timer. As police quickly evacuated the building, the local FSB explosives expert was called in to defuse the detonator; he determined that the sacks contained RDX, a explosive powerful enough to have brought the entire apartment building down. In the meantime, roadblocks were established on all roads out of Ryazan, and a massive manhunt for the Zhiguli and its occupants got underway.

By the following afternoon, word of the incident in Ryazan had spread across Russia. Prime Minister Putin congratulated the residents on their vigilance, while the interior minister lauded recent improvements by the security forces, such as the foiling of the attempt to blow up the apartment building in Ryazan.

There the matter may well have ended, except that same night two of the suspects in Ryazan were apprehended. To the local authorities’ astonishment, both produced FSB identification cards. A short time later, a call came down from FSB headquarters in Moscow that the two were to be released.

The following morning, FSB director Patrushev appeared on television to report a wholly new version of events in Ryazan. Rather than an aborted terrorist attack, he explained, the incident at 14/16 Novosyolov Street had actually been an FSB “training exercise” to test the public’s alertness. Further, he said, the sacks in the basement had contained not explosives, but rather common household sugar.

Contradictions in the FSB’s account were manifold. How to reconcile FSB headquarters’ sacks-of-sugar claim with the local FSB’s chemical analysis that had found RDX? If this truly had been a training exercise, how was it that the local FSB branch wasn’t informed ahead of time, or that Patrushev himself didn’t see fit to make mention of it for a day and a half after the terrorist alert was raised? For that matter, why did the apartment-building-bombing spree suddenly stop after Ryazan? If the attacks were truly the handiwork of Chechen terrorists, surely the public-relations black eye the FSB had received over the Ryazan affair would spur them to carry out more.

But the time for such questions had already passed. Even as Prime Minister Putin gave his speech on the night of September 23 praising the residents of Ryazan for their vigilance, Russian warplanes began launching massive air strikes on Grozny, the capital of Chechnya. Within a few more days, Russian armored battalions that had been massed on the border for months crossed into Chechnya, marking the start of the Second Chechen War.

Events moved very quickly after that. On New Year’s Eve 01999, Boris Yeltsin stunned the nation by announcing that he was stepping down from the presidency effective immediately, which made Vladimir Putin acting president until new elections could be held. And instead of holding them sometime in the summer, as originally scheduled, those elections would now occur in just ten weeks’ time, leaving Putin’s many competitors for the position little time to prepare.

In a presidential poll taken in August 01999, Putin had garnered less than 2 percent support. By March 02000, however, riding a wave of popularity for his total-war policy in Chechnya, he swept into office with 53 percent of the vote. The reign of Vladimir Putin had begun, and Russia would never be the same.


For our next meeting, Trepashkin invited me into his apartment. I was a bit surprised by this — I’d been told that, for security reasons, Trepashkin rarely brought visitors to his home — but I guess he figured all his enemies knew where he lived, anyway.

It was a pleasant enough place, if a bit on the spartan side, on the ground floor of a high-rise tower surrounded by other high-rise towers in southern Moscow. Trepashkin gave me a quick tour, and I noticed that the only space with even a hint of clutter was the tiny, paper-filled room — a converted walk-in closet, really — he used as his office. One of his daughters was home, and she brought us tea as we settled in the sitting room.

With a vaguely embarrassed smile, Trepashkin offered that there was actually another reason he rarely had work-related meetings at his home: his wife. She wants me to stop all this political stuff, but since she is away this morning… His smile eased away. Well, it’s because of the raids. You know, they came charging in here — he waved toward the front door — with their guns, shouting orders; the children were terrified. It really affected my wife, and she is always worried it will happen again.

The first of those raids had occurred in January 02002. Late one night, a squad of FSB agents burst in and proceeded to take the apartment apart. Trepashkin maintains they found nothing but instead planted enough evidence — some classified documents from the FSB archives, a handful of bullets — to enable prosecutors to hang three “pending” charges over his head.

It was their way of putting me on notice, he explained, of letting me know they would come after me if I didn’t straighten up.

Trepashkin had a good idea of what had sparked the FSB’s attention: Just days before the raid, he had started getting telephone calls from the man regarded by the Putin regime as one of Russia’s greatest traitors, Alexander Litvinenko.

Lieutenant Colonel Litvinenko’s fall from grace had been swift. After his 01998 press conference alleging the URPO assassination plots, he’d spent nine months in prison on an “abuse of authority” charge and had then fled Russia as prosecutors prepared to move against him again. With the help of the now exiled tycoon Boris Berezovsky, Litvinenko and his family settled in England, where he joined forces with Berezovsky to expose to the world what they claimed were the crimes of the Putin regime. A primary focus of that campaign was getting to the truth of the apartment-building bombings.

So this is why he was calling, Trepashkin explained. Litvinenko couldn’t come back to Russia, obviously, so they needed someone here to help with the investigation.

Easier said than done, for by January 02002, Russia had become a very different place. In the two years since Putin had been elected president, the once-thriving independent media had all but disappeared, while the political opposition was being steadily marginalized to the point of insignificance.

One indication of this chilling effect was the revisions performed on the shakiest aspect of the government’s bombing story, the FSB “training exercise” in Ryazan. By 02002 the Ryazan FSB commander who had overseen the manhunt for “the terrorists” now supported the training-exercise explanation. The local FSB explosives expert who had insisted before television cameras that the Ryazan sacks contained explosives suddenly went silent on the whole matter and disappeared from sight. Even some of the residents of 14/16 Novosyolov Street who had appeared in a television documentary six months after the incident to angrily deride the FSB’s account and insist the bomb was real now refused to talk with anyone beyond allowing that perhaps they’d been mistaken after all.

I told Litvinenko that the only way I could become involved was in some kind of official capacity, Trepashkin explained in his sitting room. If I just went out on my own, the authorities would move against me immediately.

That official capacity was fashioned at a meeting held in Boris Berezovsky’s London office in early March 02002. One of those in attendance, a Russian member of Parliament named Sergei Yushenkov, would organize a blue-ribbon committee of inquiry into the bombings and make Trepashkin one of his investigators. Another attendee was Tatiana Morozova, a 31-year-old Russian émigré living in Milwaukee, Wisconsin. Morozova’s mother had been killed in the Guryanova Street blast, and under Russian law that gave her the right to review the government’s records on the case; since Trepashkin had recently obtained his license to practice law, Morozova would appoint him as her attorney and petition the courts for access to the FSB’s Guryanova Street files.

So I agreed to both of these ideas, Trepashkin said, but the question was where to look first. So many of the reports were unreliable, and so many people had changed their stories, that my first goal was to get access to the actual forensic evidence.

Also easier said than done, for a hallmark of the government’s response to the bombings had been a peculiar haste in clearing away the ruins. Whereas, for example, the Americans had spent six months sifting through the remnants of the World Trade Center after September 11, regarding it as an active crime scene, Russian authorities had razed 19 Guryanova street just days after the blast and hauled everything away to a municipal dump. Whatever forensic evidence had been preserved — and it wasn’t clear that any had — was presumably locked away in FSB storehouses.

While what he discovered didn’t pertain to the specifics of the bombings, Trepashkin did soon manage to come up with something quite interesting.

One of the odder footnotes to the whole affair was a statement that Gennady Seleznyov, the Speaker of the Duma, had made on the floor of Parliament on the morning of September 13, 01999. I have just received a report, he had announced to legislators. An apartment building in the city of Volgodonsk was blown up last night.

While Seleznyov got the basics right — an apartment building had indeed just been blown up — he had the wrong city; the blast that morning had been at 6/3 Kashirskoye Highway in Moscow. Which put the Speaker in kind of an awkward spot when an apartment building in Volgodonsk was blown up three days later. At least one Duma member found that puzzling.

Mr. Speaker, please explain, he had asked Seleznyov on the Parliament floor, how come you told us on Monday about the blast that occurred on Thursday?

In lieu of an answer, the questioner had his microphone quickly cut off.

To many observers, it suggested that someone in the FSB chain of command had screwed up the order in which the bombings were to take place and had given the “news” to Seleznyov in reverse.

Searching around nearly three years after the fact, Trepashkin says he determined that Seleznyov had been given the erroneous report by an FSB officer, though he won’t say how he knows.

But with progress also came the potential for danger to Trepashkin. One of those who had attended the London meeting, human-rights activist and Berezovsky lieutenant Alex Goldfarb, became concerned enough about Trepashkin’s welfare that he arranged a meeting with him in Ukraine in early 02003. The two had never met before, and Goldfarb found it an odd encounter.

He was one of the stranger people I’ve ever met, Goldfarb recounted. He had no interest in the philosophical or political implications of what he was doing. To him, this was all just a criminal case. In the back of my mind, I was thinking, ‘Is this guy crazy? Doesn’t he appreciate what he’s up against?’ but I finally concluded he was this kind of supercop — you know, a Serpico figure. He was determined to do the right thing because it was the right thing to do; it was just that simple. Still, Goldfarb felt it his duty to at least alert Trepashkin to the deepening peril, the very little that could be done if the authorities decided to go after him. The more he pressed on this, though, the more Trepashkin seemed to bristle.

He didn’t care about any of that, Goldfarb remembered. I think he still believed he was fighting to reform the system, rather than that he was up against the system itself.

But as it turned out, the hammer first fell elsewhere. In April 02003, Sergei Yushenkov, the Duma member who had hired Trepashkin for his committee of inquiry, was murdered in front of his Moscow home, shot down in broad daylight. Three months later, another committee member died under mysterious circumstances. The two deaths effectively ended the independent inquiry — which also meant that Trepashkin was now essentially on his own. Still, acting as Tatiana Morozova’s attorney, he soldiered on — and in July 02003, he finally hit pay dirt. It hinged on another loose end in the case, one that no amount of cleaning up after the fact could quite tie off.

In the hours just before the Guryanova Street bombing, the FSB had released a composite sketch of a suspect based on information provided by a building manager. But soon after and with no explanation, that sketch had been withdrawn and replaced with that of a completely different man. This second man had long since been identified as one Achemez Gochiyayev, a small-time businessman from the region of Cherkessia, who had immediately gone into hiding. In the spring of 02002, Alexander Litvinenko had tracked Gochiyayev to a remote area of Georgia where, through an intermediary, the businessman steadfastly insisted that he had been framed by the FSB and had only run because he was sure they would kill him.

It made Trepashkin very curious to learn the identity of the man in the first sketch, even more so when, going through the voluminous FSB files on Guryanova Street, he discovered there wasn’t a copy of it to be found anywhere. As a last resort, he started sifting through newspaper archives to see if any had run that sketch before the FSB had pulled it from circulation. And there it was.

It depicted a square-jawed man in his mid-30s, with dark hair and glasses. Trepashkin was convinced he knew the man, that in fact he had arrested him eight years before. He believed it was a sketch of Vladimir Romanovich, the FSB agent who had manned the electronic-surveillance van for the Raduyev gang during the robbery of Bank Soldi.

Trepashkin’s first thought was to find Romanovich and try to compel him to reveal his role in the apartment bombings. Not likely. As far as Trepashkin could determine, shortly after the bombings, Romanovich had left Russia for Cyprus and died there in the summer of 02000, killed by a hit-and-run driver.

Trepashkin then tracked down the original source of the sketch, the Guryanova Street building manager.

I showed him the sketch of Romanovich, Trepashkin said in his sitting room, And he told me that was the accurate one, the one he had given to the police. But then they had taken him to Lubyakna [FSB headquarters], where they showed him the Gochiyayev sketch and insisted that was the man he saw.

With his bombshell, Trepashkin planned a little surprise for the authorities. The FSB had long since released the names of nine men they claimed were responsible for the Moscow and Volgodonsk bombings. Ironically, considering that the bombings had been the chief pretext for embarking on the Second Chechen War, none of these suspects were Chechen. By the summer of 02003, five of those men were reportedly dead, and two others remained at large, but the trial for the two in custody was slated to begin that October. As attorney for Tatiana Morozova, Trepashkin intended to attend the trial and introduce the Romanovich sketch as evidence for the defense.

He took an added precaution. Shortly before the trial’s start, he met with Igor Korolkov, a journalist with the independent magazine Moskovskiye Novosti, and described the Romanovich connection in detail.

He said, ‘If they get me, at least everyone will know why,’ Korolkov explained. He was apprehensive, tense, because I think he already knew they were coming for him.

Sure enough, shortly after meeting with Korolkov, Trepashkin was picked up by authorities. While he was being held, the FSB conducted another raid on his apartment, this one involving a whole busload of agents.

I understand it was very exciting for the neighbors, Trepashkin said with a laugh, the biggest thing to happen around here in a long time.

They brought him up on an old FSB standby — possession of an unlicensed gun — but the judge, apparently familiar with that tired cliché, immediately dismissed the charge. Prosecutors then turned to the charges they still had pending on Trepashkin from the raid two years earlier and the classified documents he maintains were planted. It wasn’t much, but it was enough; tried in a closed court, Trepashkin received a four-year sentence for “improper handling of classified material” and was shipped off to a prison camp in the Ural Mountains.

In his absence, the two men tried for the apartment bombings were found guilty and sentenced to life in prison. Declaring the matter officially closed, the government then ordered all FSB investigative files on the case to be sealed for the next seventy-five years.


My last question to Mikhail Trepashkin was something of a throwaway.

We were standing on the sidewalk outside his apartment building, and I asked him if, in looking over the trajectory of his life for the past fifteen years, he would have done things any differently.

It was a throwaway because people in Trepashkin’s position, those who have waged battle against power and been crushed, almost invariably say no: In the pursuit of justice or liberty or a better society, they explain, they’d do it all again and in just the same way. It’s what such people tell themselves to give their suffering meaning.

Instead, Trepashkin gave a quick laugh, his face creasing into his trademark grin.

Yes, he said, I would have done things very differently. I see now that one of my flaws is that I am too trusting. I always thought the problems were with just a few bad people, not with the system itself. Even when I was in prison, I never believed that Putin could actually be behind it. I always believed that once he knew, I would be released immediately. Trepashkin’s grin eased away; he gave a slow shrug of his powerful shoulders. So a certain naïveté, I guess, that led to mistakes.

I wasn’t wholly convinced of this. More than naïveté, I suspected his “flaw” was actually rooted in a kind of old-fashioned — if not downright medieval — sense of loyalty. At our first meeting, Trepashkin had given me a copy of his official résumé, a document that ran to sixteen pages, and the first thing that struck me was the prominence he’d given to the many awards and commendations he had received over his lifetime of service to the state: as a navy specialist, as a KGB officer, as an FSB investigator. As bizarre or as quaint as it might seem, he was still a true believer. How else to explain the years he had spent being the dutiful investigator, meticulously building cases against organized-crime syndicates or corrupt government officials, while stubbornly refusing to accept that, in the new Russia, it was the thieves themselves who ran the show?

Of course, it was also this abiding sense of loyalty that rather paralyzed Trepashkin and prevented him from learning from his past “mistakes,” from living his life any differently in order to get out of harm’s way. For that matter, even the change of venue of our meeting from his apartment to the sidewalk outside was kind of a testament to Trepashkin’s obduracy; his wife, returning home earlier than expected, had been so incensed at finding him meeting with a Western journalist that she’d promptly kicked both of us out of the house.

Well, what can you do? Trepashkin had whispered as we’d fled, as if he really had no control over the matter.

But perhaps his wife’s edginess that day — September 25 — was rooted in something else. That afternoon, Trepashkin was headed downtown to meet with a handful of his supporters, and then at 6 P.M. they would hold a demonstration in Pushkin Square to demand a new investigation into the bombings. You should come by, he said with his usual grin. It could be interesting.

In the five years since Trepashkin had first gone off to prison, there’d been a lot of changes in Russia — but none of them particularly auspicious for a man like him. In March 02004, Vladimir Putin had been reelected with 71 percent of the vote, and he’d used the mandate to even more forcefully restrict political and press freedoms. In October 02006, Anna Politkovskaya, Russia’s leading investigative journalist and someone who had written extensively on the murky connections between the FSB and Chechen “terrorists,” had been shot to death in the elevator of her Moscow apartment building. The following month, it had been Alexander Litvinenko’s turn to be eliminated.

But perhaps most dispiriting, it appeared the Russian public saw very little cause for worry in all this. Instead, with their economy booming on a flood of petrodollars, most seemed rather pleased with Putin’s tough-guy image and his increasingly belligerent posture to the outside world, the whiff of superpower redux it conveyed. This image was fittingly captured in May 02008 when Putin, constitutionally barred from a third term as president (although he remained on as prime minister), officially handed the reins of state over to his handpicked successor, Dmitry Medvedev. For the occasion, the two men donned matching black jackets with Medvedev in jeans, looking less like co-heads of state than a pair of gangsters as they strutted about Red Square. Even Russia’s ferocious intervention in Georgia in August 02008, an act roundly denounced in the West, spawned a new burst of Russian national pride, a new spike in Putin’s popularity.

Perhaps not surprising, then, the rally in Pushkin Square was a rather pitiful showing. Other than Trepashkin and his closest aides, perhaps thirty demonstrators showed up. Many of them were elderly people who had lost relatives in the bombings, and they stood mutely on the sidewalk holding up posters or faded photographs of their dead. The small band was watched over by eight uniformed policemen — and presumably a number of others in plainclothes — but it seemed quite unnecessary. Of the vast throngs passing on the sidewalk at rush hour, very few gave the protestors a second glance, and fewer still took the leaflet proffered them.

Watching Trepashkin that evening, it seemed there might be another way to understand why someone like him was still alive while people like LItvinenko and Politkovskaya were dead. Part of it, no doubt, is that Trepashkin has always shied away from pointing an accusatory finger directly at Putin or anyone else in connection with the apartment bombings. This fits with his criminal investigator’s mind-set: that you only make accusations based on facts, on what is knowable and certain.

But surely another part of it is his single-minded focus on getting to the bottom of the apartment bombings, his bringing the same level of dogged tenacity to that case as he did to the Bank Soldi affair. This was the problem for Litvinenko and Politkovskaya: They made so many accusations against so many members of Russia’s ruling circle that they gave their enemies safety in numbers. For Trepashkin, there is really nothing else but the apartment bombings, and if he is murdered, everyone in Russia will know why.

The irony, though, is that by continuing to push on with the case, and by continuing to call for a public investigation, Trepashkin may also be propelling himself ever closer to the answers that will destroy him. So long as those behind the bombings are confident that they have won or that they have at least sufficiently buried the past, he remains relatively safe. It is when the crowds start taking his leaflets that the danger to him grows.

That day may now be fast approaching. Amid the international economic collapse of the past year, few countries have been more ravaged than Russia, and almost every day brings accounts of new popular protests: against the oligarchs, against government policies, increasingly against Vladimir Putin himself. It may not be very long now before the Russian people start to ask themselves how all this was set in motion and remember back to the awful events of September 01999.

But it didn’t come on that day in Pushkin Square. On that day, the throngs were still true believers in the Russian renaissance, and they hurried on past Trepashkin toward the subway and home, hurried toward the bright, shiny future their ruler has promised them.

written 5 September, 02009 Comments

Transcript of ‘Rescuing GeoCities’

Transcript of the Future Tense episode ‘Rescuing GeoCities’.

Participants

Transcript

Jon Gordon: Rescuing GeoCities. This is Future Tense from American Public Media; I’m Jon Gordon. Yahoo! said last week that it would shut down its GeoCities personal website service later this year. Hard to believe now, but Yahoo! paid about three billion dollars for the company back in 01999. You may recall that GeoCities allowed users to design personal websites, but the pioneering service has long since been eclipsed by blogs and social networks.

So what’s going to become of the million-plus GeoCities homepages out there? Yahoo! is saying only that it will provide details later this summer on how customers can save their own data. Jason Scott believes GeoCities deserves saving. Scott runs TEXTFILES.COM, a site that’s devoted to computer history. He’s lead organizer for a new group called the Archive Team, which is working to save a growing body of endangered Internet content, including GeoCities.

Jason Scott: A lot of sites that attracted attention over the past fifteen to twenty years of the web have been shut down — sometimes abruptly. And we’re at a point right now where so many people are willingly putting data that they create and they own online in other locations, and then these companies, right now, feel no mandate to hold this for very long, once they decide for whatever reason to take it down. These data locations have no sense of responsibility — it’s not shameful for a company to just turn off fifteen years of community memory.

JG: What is it about GeoCities that makes you want to do this? What is the potential value in saving GeoCities?

JS: GeoCities had a reputation — though the late ’90s and certainly the early twenty-first century — of being kind of the dumping ground for people who didn’t know what they were doing: a lot of websites that were ugly, that weren’t well-written, that were boring and so on. But in point of fact, it’s a beautiful snapshot of an entire population coming online for the first time. And now that online life has becoming the norm, certainly in developed countries, it represents this turning point.

JG: So how are you actually saving all these GeoCities sites?

JS: So, we’re doing things like checking websites to see if people link to GeoCities, we’re doing Google searches to find the names of different GeoCities sites, and we’re just trying to capture as much as can. We know there’s a lot, but we’re just kinda stepping through. And my attitude is similar to if you’re trying to rescue things from a burning house, which is: you run around and grab the five or ten things or more that you can carry, and run out. And that’s not everything you own, but at least you got something.

JG: So what do you intend to do with all this material?

JS: It’s not entirely clear to me what to do with the material after rescuing it. I don’t really think of it that way. I like to be the guy who, at that historical point, at the historical point we’re in, I said ‘Let’s grab a copy’. And in maybe a year or ten years, someone will say ‘Man, I’m really glad somebody was there to do that, because it turns out this was an important piece of information — we could not have known it back then, in 02009, but here in 02019, it’s so vital that we have it.’

JG: Jason Scott with the Archive Team; more information at archiveteam.org. This is Future Tense; I’m Jon Gordon.

written 30 April, 02009 Comments

Transcript of ‘HOWTO: 149 Surprising Ways to Turbocharge Your Blog with Credibility!’, Pt. 2

Transcript of John Gruber and Merlin Mann’s SxSW Interactive 02009 presentation, HOWTO: 149 Surprising Ways to Turbocharge Your Blog with Credibility!.

Continued from pt. 1

Transcript

Merlin Mann: All too well.

John Gruber: Honestly, you cannot pay your rent with attention. I mean, I’ve tried. You can’t buy fast cars; there’s all sorts of stuff you can’t buy with it. But it has value, and you’d be surprised at what happens when it builds up.

MM: And there’s one other thing I just wanna underscore, and ’cause I’m done kissing Zeldman’s ass, I’m gonna kiss his John’s ass for a minute. There’s this one point that I really don’t wanna miss in all this ranting, which is the result of a confluence of voice — or, y’know, obsession — and what you have… Does that make sense? Do you follow what I’m saying?

Like, you’ve got something that you care a lot about, and you’re obsessed about — it’s almost like an intellectual fetish. And then you’ve got something that’s your angle on that. And to me, the more you zero in on both of those things — get crazy specific about the thing… Don’t just, don’t have a blog about Star Wars; have a blog about Jawas. Or, like, this one Jawa that’s just in the scene for a minute. Like, it’s gonna be so much easier for you to dominate, first of all; you’re gonna become the go-to guy for that one Jawa, right? And what does that mean?

Well, when something happens in the world of Apple, as is so often the case, do I go to Google News? Do I go to — actually, David Pogue’s pretty great — but, no, I go to Daring Fireball. Because John not only tells me that something happened, he tells me — he tells me what’s happened, in a very terse kind of well-edited way, with his little New York Times style guide that he’s so in love with — he tells me what happened; he tells me what it means; and then he tells me what he thinks about it. And how many people do you know who are capable of all three of those? Well, I’ll tell you what. On the Internet, there’s a ton of people that will tell you that something happened. Mostly they’re gonna, like, link from somebody else, who told them that it happened.

JG: When I wanna know, when I have, like, a collection of index cards, and I wanna know the best way to paperclip them together…

MM: So angry. So angry right now.

JG: … you know, like in a… some sort of official way…

MM: Ha-ha.

JG: … what model paper clip…

MM: Stanley Kubrick, ha-ha. No, I just wanna be clear, I admire you, but I do not like you.

JG: We’re done.

MM: Oh, is that blinking? Is it blinking to leave? … I think I just had a stroke. Am I here? Are you guys… it’s like that video for ‘One’; are you sure you guys are really here? It’s a Metallica song.

JG: You did a thing. I wanna say, I have this printed out, I gotta read this. We were…

MM: nerd voice ‘So…’

JG: … at the final stages of planning this.

MM: Last month.

JG: I don’t even know what you drove you to forward this to me, but… Somebody just out of the blue wrote to you and it was like a young kid, and he was like, twenty-two, and he was like ‘I love your site. I wanna do something similar like this, and I care about blah blah blah. What are your advices to me as a young blogger who wants to take it seriously?’

And here’s what you wrote; I’m gonna read this. And this is the greatest thing, I advise everybody who ever wants to speak on a thing like this: do a co-thing, ’cause then it’s, like, complete liberty to just steal anything that the other person has ever written.

  1. Give away more stuff than you think you should, and make it easy for people to get.
  2. Focus on diverse secondary revenue streams, and always have your eyes open for new and replacement ones.
  3. Don’t do stuff that seems profitable, but potentially messes up the reason people like you.

And that… in three items, that is so exactly what I think is the right way to do this, in a way that you will be surprised at the opportunities they present yourself.

MM: Thanks. I hope it helped. And the thing is, again, it’s just so important to underscore that, like, I’m not just sandbagging. Like, I don’t think I do all this stuff great all the time, but here’s the thing: there’s very little to lose at first, when you start doing this stuff. ’Cause yeah, you’re doing it in public, but you also have time to figure out what it is that you’re doing.

And in fact, there’s a certain obligation you have to constantly re-figure out what you’re doing. Right? Because it’s easy enough to figure out how to do one thing once, but to have a long live career in this stuff you’ve gotta figure out how to do it over and over and over. And I just think these are patterns that make sense. The ‘giving away stuff’, this is where, let’s be honest, this is where we’re so much smarter than corporate America. Let’s be honest.

JG: It’s the opportunity. I mean, we’re the mammal —

MM: Giant.

JG: It’s mammals versus dinosaurs.

MM: It is. It is. It’s like, how many ways can I, like, figure out how to make this hard for you to do, and then not make money out of it? And it’s mind-blowing to me, when I’m like, y’know, how about a world where you decide that you’re selling ideas, rather than plastic or paper? Yeah, y’know what? If you have enough great ideas — that people steal, whatever that means — well, if you’ve got enough great ideas, then people will wanna buy your paper and plastic. But if you start out by going ‘I’m a merchant of paper’, or ‘a merchant of plastic’, nobody hears that you sell paper and goes ‘Oh, that’s for me.’. Right? And so to me, you go, like, ‘Damn right it’s free!’.

I said this recently at this panel in Atlanta, but… in 02007 I did a talk at Google. I went and I showed up with a computer and I talked. And I talked about email, to some people who really needed to hear about it. You were there, Greg Veen was there — hi, Greg. And I said, here’s the stuff you should do with email and it’s this thing I do called Inbox Zero. And the last time I —

JG: Service mark, Merlin Mann.

MM: Service mark. I did actually get a service mark, yeah. Hired Arrington. The… sorry, just kidding Mike, don’t kill me. I love my daughter.

I did that… anyway, I’m not trying go ‘yay me’, but, like, I went in there that day just saying ‘Okay, this is cool. I’m gonna hang out with Veen, and, like, this’ll be fun.’. And the last time I checked on Google Video… people didn’t watch the whole thing, but, like, it had started to load at least 400,000 times. And now people say ‘Hey, come tell this to my company.’. Right? Like… And if I had, just like I say, like if I had gone ‘No, you’ve gotta pay a nickel to watch me talk about email on the Internet.’, would you have done it?

Like… some of you know Inbox Zero, right? Some of you have heard it or seen it, like, or are sick of hearing about it? But, like, you know it? And why do you know it? You know it because it was made embeddable, and anybody could go… What does embedding a video on your website mean? I mean, sometimes you’re making fun of stuff, but most of the time when you embed something, you’re saying ‘This is something I relate to.’. And if you keep that sealed in a little jar, and then make people pay just to see the jar, let alone what’s inside of it, that’s mind-boggling to me.

You, at one time — I was giving you shit about this — at one time, you had a membership model where if you wanted the full RSS feed, you had to be a member.

JG: Right, I was…

MM: Was that easy to maintain?

JG: Well, no. And the craziest thing about it is — how many people here use the Google Reader? Yeah, look at all those hands.

MM: Single biggest source of traffic.

JG: Right. Google Reader is huge, and everybody Google Reader. Here’s the thing that was so fucked up about about my ‘You have to pay to get the RSS feeds.’: it was not, like, a supply-and-demand problem. I did have plenty of readers who were like, ‘I am happy to pay.’. And my idea, my thinking was, ‘I might wanna put ads on a website, but ads in RSS, I dunno if that’s gonna work out, so I can’t just put everything in there for free. I’ll charge twenty bucks a year.’.

Here’s the thing: the feeds didn’t even work with Google Reader, ’cause Google Reader doesn’t do, like… I had authentication to, you get, like, a username and password. So the single most popular reader didn’t even work with them. In hindsight, it’s like, ‘Oh my God’, I mean, I needed a smack.

MM: But at the time, ’cause right, you guys have done this — some of you are entrepreneurs and business people and people who try to make money on things — your first thought is, like, this panicy lizard-like, again, lizard-brained idea of, like, ‘Ah! How do I make a little money of of this?’

And you end up, it’s almost like going to Safeway, and if Safeway is gonna give you a free sample, but they’re gonna put way too much salt on it, ’cause you’re a deadbeat, y’know? It’s like if you don’t give stuff away and let people figure out why you’re awesome, why would they ever be interested in anything that you do? And if you don’t have the confidence to go, like, ‘My ideas, and the things that I have to say are so valuable that, like, I’m not worried that I’ll run out of them. I’m not worried that there’s any scarcity to what I have to say about this.’ So yeah, people scrape my RSS feed hundreds of times a day. But that’s not me; I’m not my RSS feed. I’m the ideas that went into the RSS feed.

JG: So I think one of the things that is so frustrating, for me, to watch people who just don’t seem to be taking advantage. I mean, it’s, again, so trite, but the Internet is awesome. It is totally fuckin’ awesome. You can do anything. And the thing that is so amazing is that it’s not just —

MM: Write that down, maybe Twitter that.

JG: Yeah, credit to me.

MM: nerd voiceRT @gruber’…

JG: — is that it’s not just that we all have a printing press now, and now we can do the same thing that big media companies with big printing presses and Teamsters and trucks that they can deliver their stuff can do; it’s that we can actually do it better, we can do it in ways that actually make people happier. It’ll make people happier to read stuff on my website or your website, where it’s just not even, it’s not all crapped up, and it’s, it’s just honest and it’s plain written, and you can just have it, or… Jonathan Coulton, you can just go to his website and the music is just MPEG-3s, and you just, y’know, give him some money, and just download some stuff.

MM: Yeah, ‘If you wanna put it in a movie, fine; give me credit and put it in there.’

JG: Right.

MM: Right. Or, y’know, like, RSS? It’s so amazing to me that, like — we should get along on this next point; we’ve got ten minutes — but this whole thing of, like, ‘I need you to do it this way, or I’ll be sad.’. Y’know, it’s like, y’know what? I don’t care, if you… Print it out. Like, oh my God, I’m so glad you’re even a little bit interested in this. Put it on Kindle — thank you, Marco. Do whatever you wanna do with it. Like, y’know? But do something with it.

And, like, it’s hilarious to me, especially, when people are like, y’know, they’ve got this very small amount — it’s not funny to me that they aren’t successful, but it’s funny to me that people get so torqued up about all this IP stuff when nobody cares what they’re doing. And you look at the people who the confidence to go ‘I’m a giant, successful…’ — like Jonathan. Jonathan’s incredibly, Jonathan Coulton is incredibly successful precisely because he’s given it all away. So, this is a really douchey one, so let’s do this fast.

JG: Yeah.

MM: I said this at this public media conference, and I really believe this: don’t become too obsessed with the thing you’re determined to make money on. And for most folks, that’s ‘I wrote something and now I need to make this much money on it.’. And if you’ve got a pro blog, and you’re paying people to do multiple posts a day, or whatever, you need that kind of ROI. Your Excel has to line up. But if you’re a personal publishing person, I think it’s really valuable to say ‘I’m gonna keep my ears open. Maybe…’. And you know, you need to figure out what you can live with. Like, is an Amazon store okay? Is selling links okay? What am I gonna do, right?

JG: Well, and —

MM: Am I gonna have, like, a little store where you can buy a camera?

JG: And stuff that didn’t work starts to work. Amazon stuff never made me more than, like, I dunno; $10 a month — and then all the sudden it started making me real money. I mean, I dunno what I did differently, but then, y’know…

MM: And again, you’ve got… so you can make a boatload of money on Amazon, but you also have to weigh the extent to which people go ‘You’re being kind of a dick with the Amazon links.’.

JG: Right.

MM: And you have to listen to when people are saying ‘enough already’. Right? And what’s the last one? Oh, that one, yeah. You wanna do this one?

JG: Yeah, this is a good one.

MM: Yeah, ’cause this is you, dude; number three is you. This is all you.

JG: Don’t do stuff that seems profitable, but potentially messes up the reason people like you… That’s you too. I mean…

MM: Not really. No, I, God, I’ve done so many dumb things. I still do so many stupid things, and it’s like, it takes me a while to figure it out — I’m having a cookie, I hope you don’t mind. That was a good cookie.

JG: But that’s like crummy text links, and…

MM: Yeah. I sold text links on my site for a while, and I didn’t feel good about it. It made, like, pretty good dough, and I… sometimes now, just quick sidenote, like, I still kinda can’t believe people do that, just because, even if you don’t care about breaking Google, it’s just kinda surprising, but, like, I did a thing where — I do a lotta my reading where I’ll throw stuff into Evernote to read later, and I’ll do that, like, I’ll strip out all the CSS and just throw it in Evernote — and there was a site, somebody who’d written for Kevin Kelly, I went and visited Cool Tools, I went and visited his page, in plain text, or in, y’know, unstyled? It was totally, like, all poker and Viagra links at the top of the page.

And so, like, this guy, who like — and I’m not trying to call the guy out; I’m not trying to, like, shame him, ’cause that’s his decision — but what did his decision mean? His decision meant that for me I went ‘Wow, I’m not sure I’m gonna read the rest of this.’. ’Cause, like, that’s not… eh, I dunno, for where I am in my life right now, that’s kinda not cool. And so, like, I dunno.

And again, I’m not trying to shame him, but I am saying — I know you do shit like this all the time. I know that there’s all kinds of people that you just won’t even link to because they’re, they’ve got the Kubrick theme running, and they haven’t even tried.

But y’know, do you know what I’m saying, that you go somewhere, and you’re just like, wow, there’s just too many ads on this page, or there’s something. And you have to figure out for the folks you’re trying to reach, how do you find the balance of making a little bit of dough, but still not crushing the bunny, but not having the person who might give you a hundred thousand dollars visit the site and go ‘Wow, this guy sure is interested in poker and Viagra.’.

JG: Well, I’ll give you an example that I really hate, and you have to be careful about, it is with sites that do, like, a lot of, like, ten links a day about Mac news, and they will find something interesting, and instead of just sending you there to read it, they will do their best to summarize it. And sometimes the summaries — the ones that I call out; I have called out a couple over the times — where their summary is longer than the thing that they were linking to originally.

MM: And then the link’s after the jump.

JG: Right, and then the link is after the jump. And the whole point, of course, is to just sort of steal the idea and not even send you there. And of course it seems profitable —  because it is, you’re getting the pageviews for it, and then you’re trying to get other people to link to you instead of the original site — but in the long run, I think it’s terrible, because I think readers know, I think readers eventually find out, they’ll notice, ‘Hey, that wasn’t even theirs.’, and then your credibility is gone.

MM: We should start kinda wrapping up. We only have time for a couple questions.

JG: Or one more, one quick more, is the pagination thing. And that is something —

MM: Oh, dude, the pagination thing.

JG: Right.

MM: Wow, that’s awesome.

JG: I mean, I Twittered this last week, so this might be a repeat for some of you, but I said…

MM: This is good.

JG: ‘I think I’m developing a form of dyslexia. Every time I see “next page”, I think it says “stop reading and close this tab”.’ And the worst part is, if you talk to work at real, real sites — like real newspapers, real magazines; not shitbag sites; like, good sites — they know, they have the stats, they know that almost everybody stops reading an article when they get to the first ‘click here to go to the next page’. They know that they do it. And they do it anyway, because they get, like, a half of a penny for everybody who does click.

MM: Right. So, that last koan thing?, the corrupting influence of choosing a business model that doesn’t support the way you like to roll. So if the CPM model ends up corrupting the way that you wanna treat your audience, then you have to be circumspect about that.

We should probably wrap up, but just a quick, I think one thing, John, we didn’t talk about this, when we were talking earlier, but, like, something we talked about initially was talking about the kind of continuum idea. You remember that? And just this idea of, like, this is not, yeah, we’ve got a strong opinion, you should have a strong opinion too. But ultimately a lot of this stuff is just about iterative decision-making. Like, coming out of the box, always trying to work really hard to do something good is just a good idea, but then as you evolve, and as what you make evolves, I think also, I dunno, just being open to kind of, like, I dunno… am I making any sense? Does that make any sense at all? No? Really?

JG: Yeah, it’s all right.

MM: Yeah, it’s okay. It does matter, though. It does matter. And it matters to connect the people that you really admire and respect. And so I think figuring out how to do that in a way that gives you what you need is just about making those decisions. And there’s nothing that’s absolutely wrong — I mean, except for a few things — always, always, always be linking. And there’s a few things I think, but… generally, like, whatever you decide to do, as long as that supports what you wanna do, just try really hard at it. What do you wanna close with? Any advice?

JG: No, I think that’s it: try really hard.

MM: Anybody question? One question, one question. Anybody question? Somebody go to the mic. Hey, it’s Remiel!


MM: Hey, uh, hi, it’s me, Merlin, back here at the podcast. I know that right now it’s really hard to hear, but that’s our friend Gabe — who’s Remiel on Twitter — and he’s really cool. And right now, he is asking us a question about how we use Twitter, and then we answer it.


JG: I kinda use Twitter as the, like… All right, you could say, if Daring Fireball is anal-retentive, it’s like the little punctured hole in the back where all the shit just flies, ’cause I don’t care. And it’s, it’s, y’know… I’ll spend six weeks posting nothing but stuff about Sarah Palin and her family.

MM: Oh, that was a rough time.

JG: Oh, that was very popular.

MM: You were one of the many I had to take a little break from.

?: Merlin, you’re on a different boat. ? thoughts about it.

MM: Yeah, I mean, that’s what’s interesting about Twitter. Michael Lopp, you’re the one who said this, I think: what’s great about Twitter is you only really have to see things you don’t like if you wanna see things you don’t like. If you just follow the people you really like, that’s the way to roll. Twitter for me is 140 characters of id, and I’ll own that.

But, like, seriously, go out and have fun, but make something really kick ass, and, like, try to really impress the people that you love. We have been…

JG: John Gruber

MM: … and Merlin Mann.

JG: Thank you for coming.


MM: And there we were. Sorry, that was a long hour, but I… Did you like that John? That turned out pretty good, don’t you think?

JG: I think it turned out great. I usually hate my speaking appearances in hindsight. I listen to them or watch them, and it never turns out at all like I imagined. And this one, for whatever reason, it seemed like we really hit it.

MM: It coulda gone worse. I feel the same way for me; it’s like taking a bandage away, and having an idea that there’s gonna be a gunshot wound underneath it. And then just kinda poking at it, and discovering it really does hurt a lot. That’s, y’know. Pretty much everything I’ve ever done that’s how I felt.

JG: Yeah, or maybe you go in to have, like, some cosmetic surgery, and you think you’re gonna get, like, a Brad Pitt, and you take off the bandage, and it’s Andy Dick.

MM: Yeah, well, there’s a little dick under most of my bandages. We should go, but listen, I wanna say special thanks to everybody who came out. To be honest, y’know, all the folks who said nice stuff, it’s been really cool. And I wanna especially thank our pal Dave Gray, did some wonderful drawings while John and I were talking, and on the post that’s associated with this podcast episode you will see some of those, and I will point you to Dave’s site; I encourage you to check out his work. Anybody you wanna acknowledge, John?

JG: Yeah. We should thank Hugh Forrest from South by Southwest. I mean, the guy does a fantastic job. And he’s everywhere. I mean…

MM: He’s tall. He’s very calm?, and he’s tall. It’s inexplicable.

JG: I almost suspect it’s one of those gags, where maybe he’s like a twin, and there’s two of them, and they never, they make it seem like it’s one guy? Because he’s everywhere. And whenever you actually, like, need help, he’s right there, and helps you out. So thanks to Hugh.

MM: Totally. Oh, and did you wanna thank your pal Arrington? Did you wanna say…

JG: Oh, yeah, thanks to Mike Arrington for having the grace to ?

MM: And you haven’t had any problems? You haven’t found any, like, dead cats on your doorstep or anything?

JG: No. You know what’s funny though? You brought that up in the thing, and I got confused with that part. You said Mike Arrington, and I kinda went off; I thought we were talking about Nick Denton.

MM: You confused Michael Arrington and Nick Denton?

JG: Yeah, I get my dirtbags confused.

MM: Ooh, you know, they’ve both got pretty big heads; that Denton guy, he’s got a gourd on him.

JG: Yeah, it’s sorta like a, yeah, it’s like a Humphrey Bogart-type thing. You know, Humphrey Bogart’s head was about four feet tall.

MM: And he had very very small feet. Did you know that?

JG: Yeah; very small feet and an enormous head. And it made him look fantastic in black & white.

MM: He cried at his own wedding. I always found that very moving, y’know. That’s a guy who really cared. Smoked a little too much, but a good man.

All right John; well, if anything happens, if you need a witness or anything, let us know. ’Cause now you got two powerful blog guys with giant heads after you. So watch your ass. Yeah.

All right, well listen, thanks for a lot everybody, and many thanks to John; please go to his site, Daring Fireball dot net, if you’re not going there already; it’s a tremendous lot of fun.

This was super-fun, I think, for both of us to do. So thank you very much; this is 43folders; until next time, I’ll see you in cyberspace.

written 27 March, 02009 Comments

Transcript of ‘HOWTO: 149 Surprising Ways to Turbocharge Your Blog with Credibility!’, Pt. 1

Transcript of John Gruber and Merlin Mann’s SxSW Interactive 02009 presentation, HOWTO: 149 Surprising Ways to Turbocharge Your Blog with Credibility!.

Both Gruber and Mann have written follow-ups — Gruber’s is unsurprisingly the longer of the two, while Mann’s has the audio (and comics!).

Note: There is some language involved, which I have left uncensored.

Before we get started

I was forced to break this transcript into two parts because when I fed the entire thing into Tumblr the text was simply dropped. I apologize for this; I really dislike pagination (and it even comes up in the talk).

Participants

Transcript

Merlin Mann: This is 43folders, and I’m Merlin, and this is my pal John Gruber, from Daring Fireball dot net; how’s it going, John?

John Gruber: Good, how are you, Merlin? Good to be here.

MM: I’m doing extraordinarily well. This is — boy, this is really good for our first take, isn’t it?

JG: Yeah, it’s great. One taken out.

MM: That’s all we need. Well, anyway, you all know John from Daring Fireball. We did a talk about a week-and-a-half ago, at South by Southwest in Austin, and we talked about blogging. Do you remember that?

JG: I do.

MM: Yeah. We talked about trying to do a better job with your blog, and kinda doing your best, and trying to move beyond, like, a quick buck, to seeing kinda longer-term opportunities. Is that roughly what we talked about?

JG: Yeah, I think big picture it was about, maybe, turning yourself around from having your primary goal to be making a bundle of money — which probably isn’t gonna happen anyway, and really leads you the wrong way — and instead turn around, and just think… find your obsession, and follow it.

MM: Yeah. I totally… and, y’know, the response to it’s been really good, which has been kinda exciting, and so… we wanna share that with you today. So, we stole the audio straight off of the site at South by Southwest, so we’re counting on our friend Hugh Forrest to run interference with Legal; it runs just about an hour, and we’ll have a couple comments after you listen to it. For our audience’s sake, John, do you remember what the title of our talk was?

JG: Sure, it was very short and punchy. Title was: ‘HOWTO: 149 Surprising Ways to Turbocharge Your Blog with Credibility!’


MM: First of all, I cannot believe that somebody let us have this title. John and I do a lot of ambitious things that we’re pretty sure won’t turn out, as you know, and we figured somebody would change that. Don’t you think, kinda?

JG: 100%. It was, might as well have been titled ‘Title to Come’.

MM: Yeah. Yeah, T.K. Gump, yeah. It’s funny, because we pitched this months ago, and as the title implied, we were prepared to come in and provide a Tour de Force middle-aged–white–guy rant about how all of you are doing it wrong. Because… is that a core competency? Is that fair?

JG: It’s pretty much all I do.

MM: Okay. And just to be clear, it’s not, John’s not gonna tell you how it could be a little better — he might get to that toward the end — but he’s mostly just gonna tell you you’re doing it wrong.

Know what’s funny is, so we’re like ‘Hey, we’re gonna do this thing, and we’re gonna be like “Ah, don’t make a shitty site, where you’re just trying to get on Digg. Like, it’s great to be on Digg, but, like, quality, maaaan, we’re great!”’ And then, what, a couple months went by, right?

JG: Right. Well, when we got started it… When we got started the general economic situation was pretty much like a constant series of news articles about ‘Are we in a recession? Is this technically a recession? Y’need 0.9% decrease in growth over ten months, or blah blah blah’, and now, y’know, it’s ‘Are we in a depression?’.

MM: Yeah, it got to be… of course, we hadn’t done anything on this at all, for… let’s just say we, let’s say it was done, what, a month ago, we were all done?

JG: Exactly.

MM: But for a long time we didn’t do anything, and finally, it’s like ‘Can we afford the flight?’. ’Cause, yeah, I’m glad that I can educate you on how you’re doing it wrong, but I don’t know if I have enough money in the bank.

It’s been a weird trip; there’s a time that John and I have lived through, that I think — at least in my own mind — I’ll remember as the golden age of the one-person boutique personal publishing empire. And if you were lucky enough to accidentally land on that gravy train, for lack of a better word, it was pretty, it was kind of an interesting ride.

But… I think the original idea of what we wanna talk about in the end has very little to do with any economic indicators. Fair enough?

JG: Totally correct.

MM: Yeah. It is a different talk. I think there are four assumptions that we wanna toss out, just to frame this, very quickly. We’re not here to tell you what you should make, and we’re not here to tell you how you should make it, or what your ultimate goals should be — but we’re gonna assume four things about everybody in here. Step zero is we’re gonna assume that all of you make things. Right? Like maybe have a job, and you type in Excel, and you’re not a painter, and you don’t have a beret. But we’re going to assume that you make things. We’re gonna assume that you care very much about certain issues or topics, to a point where you’re really verging on obsession. We’re gonna assume that it’s important to you — whether you’re a writer, a photographer, or an interpretive dance choreographer — that you wanna get better at it, and that it’s valuable to you to use a platform like personal publishing to become a better writer, thinker, photographer, whatever. We’re gonna assume that it matters to you to have the credibility and respect of people you admire. And, for the sake of argument — just for fun, to make it worth the flight — we’re gonna assume that you would not mind making a little bit of money, or finding some kind of an opportunity that goes beyond the strictly self-improvement aspects.

JG: And so there’s this quote that I’ve sort of hung this whole thing on, right from the outset, from Walt Disney, and it’s, to me, it’s the thing that made me wanna do this talk. And he said: ‘We don’t make movies to make money. We make money to make more movies.’ And I think that’s so profound, and to me, it’s not about a subtle difference in strategy; it’s a fundamental, you’re either going this way, or you’re going that way. And so if your interest is making money, and then you decide, ‘All right, I wanna make a boatload of money from a website, how do I do that?’, well then the next fifty-seven minutes or whatever are gonna be useless to you, because that is not what we’re here to say. You can, y’know, you can get up, we won’t mind, go see Jeffrey Zeldman —

MM: Yeah, definitely go see Zeldman. That guy’s smart.

Yeah, cause I mean, I think the one thing I really regret, I think, about that kind of funny title — it’s kinda funny, right? sorta? yeah, it’s a little funny, yeah — the part about that I wanna make sure we don’t misconstrue in the desire to be a smartass is that, like I say, I’m not trying to tell you what you should do, and I’m not trying to judge anything that I just isn’t, that I would say just is not for me. And I think the dirty little secret of what we’re trying to say ultimately is that it shouldn’t matter to me. You shouldn’t care. If I’m not in your target audience, if I’m not the person you wanna reach, we should both be totally cool with you not caring what I think. And that ultimately, the people that I admire — and I think that John admires — it’s not about arrogance, but it is about having the confidence to know what you wanna say, and who you wanna say it to. And whether you wanna talk about having a good run on Twitter, or whether you wanna talk about 135 exciting new ways to launch Firefox, that’s your decision. But we wanna help you do the shit out of that, in a way that’s super-high quality. And I think that’s kind of where we’re heading. Fair enough so far?

JG: Yeah, I think so.

MM: Um… this is a little rant of mine that I’m gonna quickly go through. I have this theory… I dunno if you wanna call this blogging, personal publishing, insert-your-favorite-gerund for putting things on the web that you made. There’s a controlling metaphor for this that means a lot to me, and it’s… I tried to write this down in a way that’s clear, so I’ll read this.

Topic times voice. Or, if you’re a little bit more of a maverick, obsession times voice.

So what does that mean? I think almost all of the best non-fiction that has ever been made comes from the result of somebody who can’t stop thinking about a certain topic — a very specific aspect, in some cases, of a certain topic. And second, they got really good at figuring out what they had to say about it.

And if you have obsession without voice — or topic without voice — what do you have? You have basically a keyword search. You have pointless reblogging. You have — ah, I should say reblogging without curation, right? You guys know what I’m talking about? We all love these sites; we all, we enjoy going places where we’re seeing things that we’ve seen before, about… steampunk. And not a problem with steampunk! Steampunk’s… but, y’know, do it well.

And then, on the other hand, if you have voice without an obsession, you get a lot of, y’know, people commenting on the Thai food that they just had, on Twitter.

And I guess what we’re saying is to figure out where you are, as somebody who wants to get better. We’re assuming you wanna become like a lion of this stuff. And I guess what we’re saying is we wanna help you figure out whatever it is that you wanna do — and whatever outcome you wanna see as a result, how do you use a platform like personal publishing to become really great, to really become kind of like the go-to person for whatever the topic you’re obsessed with is.

Oh, here’s the line I wrote this morning, even though this was done a month ago. I said, whatever your topic is, try to figure how to be better at it than 80% of everybody else in the world. I… I think that’s very ambitious, but I’m gonna say that’s — and you know what, you probably won’t be. Right?

JG: Right. And I mean, I think that there’s… It’s almost like we need to warn you that there’s a certain inherent douchiness to what we’re doing, as we’re sitting up here and sort of, in some ways, holding ourselves up as the examples that are doing it right. And that’s, y’know, it’s sort of un-humble and not really what you’re supposed to do, and y’know, to be a humble, nice person. But…

MM: We’re trying, right now.

JG: And that’s really the thing; for me —

MM: That’s the 80%, is the trying, really.

JG: — I’ve got this thing, where what I write; I had this idea at the beginning, and I’ve always liked the New Yorker magazine, and it’s, just because it’s so well-written, and they will just take any topic that whoever’s writing about and go into such great length about it, even if it’s just one tiny angle of it, and they’ll just go… y’know, if they need six thousand words for it, it’s six thousand words, and it’s just so well-written.

And that just is like in a nutshell, when I wanna remind myself why I’m doing it at Daring Fireball, is I want to write about these topics I’m obessed with — and I just assume you guys know what those topics are, I don’t have to rehash them — but… if they were gonna be covered in the New Yorker, and if they were gonna pay me to do it, how would that be? How hard would I have to work to do that? And you might be saying ‘John, I like your stuff, that’s why I’m here to see you; I love your site, you really write about some of this UI stuff; and oh, tabs, that thing about the tabs — ’

MM: Tabs, yeah…

JG: ‘ — in Safari, oh, that was great! —’

MM: This is a man who cares about tabs.

JG: ‘— Oh, what was fantastic, but let me tell you something, buddy, that shit is not from the New Yorker.’. But that’s exactly it! I would be the first one to tell you that I’ve never hit that mark, and that’s…

I think it’s so important to have a goal that’s out there that you know is beyond your reach, so that you’re always improving. I do feel, I feel that I am such a better writer now than when I started the site six years ago. I mean, there’s just no doubt in my mind that I’m better at it. And I still feel like I’m nowhere near as good as I wanna be. I can write something and it’ll be the article that, y’know, when I meet people at a place like here, and they’ll remind me, they’ll say ‘I love that thing you wrote a couple weeks ago’, and it’s something that I just think, ‘Oh my God, that is so far short of the idea I set out to write, but thank you so much for saying it’, but that to me is the whole point, is that you’ve gotta have a goal that is so far out of your reach, and… it seems to me that almost everybody else is setting their goal to write…

MM: … write on a very broad topic that a lot of other people cover to a very large audience that they they don’t really care about.

JG: Right, and they’re —

MM: Some, some people; not everybody. But there’s… if everything is what you wanna do, then you’re not really doing a thing. If you wanna make everybody listen to something you have to say about everything all the time, how do they know it’s for them? How do you know that you’ve reached the right person, if you’re trying to please everybody? And, y’know, for me, I’ve got another metric that I use — I like John’s; I mean, I’m not the writer John is, and I aspire to be — I’ll take it in a slightly different direction. Y’knoww, John’s very anal retentive; I don’t know if you follow what John does, but I mean, I don’t know if you know on Twitter, John’s wife, Amy Jane — who you should follow, is the funniest person on Twitter — and she’ll just talk about how he, for like half an hour, he’ll talk about kerning, like on something in a commercial, ’cause he really really really cares about that. And you know what? Almost none of you care about that, and that’s okay with him. But for those of you who also really really really care about kerning, it’s nice to find somebody who’s on the same page.

And if John decided instead that, like, he wanted to suddenly branch off into something that he didn’t really care about because he thought it would get him a lot of pageviews, he’s gonna cut into muscle, by not caring about what you both already care about. And so, for me, I have to be honest, I aspire to different kinds of things; I’m definitely all over the map, and I’m kind of desperately always trying different things to figure out if this is the thing I wanna do, but one thing I do, I think about it differently.

I think about it in terms of, the way I put it is — you guys okay? I think about it in terms, the phrase is, ‘Who do I wanna delight?’. I try to think a lot, less about, like, ‘Is this something that will, y’know, get me this kind of link?’, and more like ‘Is this something that John would think is not a piece of crap?’. ‘Is this something that, like, if Zeldman saw it, or if Dave Gray saw this, like, would Dave be into it?’ Like, if it’s something funny, like, Adam — lonelysandwich — doesn’t think much of anything I do is funny, but I aspire to make Adam Lisagor laugh.

Do you follow me? Can you think about, like, one face behind your monitor that you see when you’re making something? Like, can you tell, like, whether you’ve made something that would make somebody’s day? Or are you just thinking about a big pot of people who will click on your stuff? Because the truth is, once you figure out who those faces are, it gets a lot easier to make something that you’re really really proud of, regardless of what it is that you wanna make.

JG: Even with something as absolutely stupid as jokes you publish on Twitter. It is true. And you do get a certain kind of feed back, like with the Favrd, or ‘fah-vard’ — how do you…

MM: I say ‘f’vard’.

JG: ‘F’vard’? Like… that’s like the elephant, isn’t that the elephant..?

MM: Oh yeah, F’vard, he’s the one with the crown.

JG: Right. But the weird thing about that is it is true. And we cheat — I mean, we are the worst cheaters in the world at Twitter, because we —

MM: We use Wikipedia and a dictionary.

JG: Well, and we wrote, like, scripts to, like, add subscribers and autofollow…

MM: unclear SEO thing…

JG: … ungodly subscriber counts and so of course we get on Favrd for really bad jokes. But to me, I get a thrill when someone who… to me, it doesn’t matter how many people say the joke is funny; but if there’s somebody who I really like who did, oh man, that is the greatest. Even if it’s like…

MM: Oh, totally. Like, if you’re watching Favrd to see who favorited your stuff, like… I mean, it’s nice, I like it when a lot of people like something, but like, when — God, why am I kissing Zeldman’s ass so bad? He’s not even here — but I see his little funny orange icon come up, and I’m like, oh my gosh, Zeldman thought something I did was okay. And yeah, that’s needy; I’ll own that. But I dunno, I think that’s meaningful.

There’s this… I think it’s Stephen King; I know it’s Stephen King, but I think the phrase he uses is ‘ideal reader’. There’s this book of his that most people are sick of me talking about, called On Writing, that I like a lot. There’s two kinds of people: there’s people who groan because you talk about On Writing, and there’s people who’ve read it and go ‘It changed my game.’. Whether you liked Carrie or not. But he uses this phrase ‘ideal reader’, which is, for him that’s often his wife, or the first reader, if you’ve ever heard that phrase. And again, I’m talking about photography, I’m talking about music; whatever you make. Like, who are you making it for? Who’s your ideal reader? Who’s your ideal reader, John?

JG: My ideal reader is like a second version of me. Like, I just imagine — no, I do! I imagine —

MM: You’re gonna go blind.

JG: I, y’know, I’m just up here to lay it all out. I’m gonna be very honest. And it is… it’s totally…

MM: We’re pushing the douche button, but I think that’s gonna be okay.

JG: I mean, but that is why, it’s like someone in my racket who’s doing the whole thing from home, most days I don’t wear pants. ’Cause there’s a lot of touching yourself involved.

MM: Ya gotta find your comfort zone.

JG: But it is. It’s me, in my mind it is that I had the idea to do this thing, the thing that has become Daring Fireball. I started it in 02002, but I had the idea long before that. And it just never seemed quite right, never seemed quite right. And I’m not sure what happened, but at some time in 02002 it seemed like, okay, I’ve gotta actually do it and try it and start it.

But in my mind, there’s, like, another version of me that is still thinking now in 02009 ‘I oughta do that site where I tell everybody how they’re wrong about everything.’. Do my little grey background with the white text because I think it looks better, and not have any crap on the page, and all these ideas. But there’s a version of me that still hasn’t done it, and he’s out there, and he thinks about the same things I think about, and he wishes that people would write about these things in great detail, and that’s who I write for. I just imagine him out there, and he just loves it.

And maybe that’s, like, the worst thing possible, ’cause that’s the thing that’s keeping him from actually doing his own site, because my site is so spot — ‘Oh, I wish I’d said that! Oh, I wish I’d said that!’. And I just keep trying to get that. And I always think too, about, like, is he out there thinking ‘Why hasn’t Gruber written about blank yet?’? Because I know he’s thinking ‘Oh my God. He’s gotta have a story in the works about whatever.’.

MM: You said something that I’ve seen quoted in other places — I dunno where you originally said this. One time we were talking on the phone and you said this, and I was like, y’know…

At a time when it was considered de rigueur to have comments, and I still had comments on my site because I thought I had to have comments on my site, I was like ‘You don’t do comments. What’s the deal with that?’ And you, like, you said — you probably remember what you said — and you said it in this kind of passionate tone, and it was kind of scary — and you were like ‘I wanna own every single pixel on my site, from the top left to the lower right. And if I have somebody come in — even if it’s somebody incredibly smart; even if it’s whoever; even if it’s SeoulBrother comes in and has something to say, like somebody really smart and really funny, like, it’s not my site any more.’.

Well okay, so should you turn off comments? No. That’s not what we’re saying. But we are, I am saying, figure out, if you do decide to own every pixel of what you make — and I’m not saying I do; I’m pretty slack about this stuff — but I think it’s a good pattern, if you’re thinking about this stuff, to figure out how you own every pixel of what you’re making, to the point where you know you’re reaching who you wanna reach. If it’s a broad audience, that’s not a problem. Reach the shit out of a broad audience.

But if you’re sitting there going, like, ‘I really hope Malcolm Gladwell sees this someday’, or ‘I really hope Anne Lamott sees this one day’, or, God love me, ‘I really hope Stephen King sees this and thinks this is smart, ’cause that would mean a lot to me’, then that that gets you thinking in a really different way from ‘I just need to post twelve times a day.’. It’s a very different approach.

JG: I have a good story about that sort of thing, where you finally find out that the people who you hope are reading and enjoying your site are actually reading it.

And this was two years ago at WWDC, the big Mac nerd development conference, and it was the first one after they had announced the iPhone; the iPhone wasn’t even actually out yet, and so you couldn’t actually have an iPhone, but everybody already wanted to program for it at WWDC, ’cause they’d already said that it’s gonna be Cocoa, and Cocoa programmers, as soon as they hear that, they just get a big stiffy, and… all they wanted to do was write iPhone software. And at the big announcement with Steve Jobs and Scott Forstall — the guy who’s in charge of this SDK stuff — they come out, and they, quote, they say ‘We have a really really sweet solution for all you guys who wanna write software for the iPhone.’. And it was: you can write web apps and they’ll run in MobileSafari.

And it was so exactly the — the four thousand people who go to WWDC are the four thousand people out of the six billion on the planet who least wanted to hear that. And, so, the consensus, it was, like… Those announcements go out they’re, like… they’re really for the press, ’cause those things go out and USA Today writes about them and David Pogue writes about them in the New York Times and millions and millions then read what Pogue wrote in the New York Times.

But for the four thousand people in the room? I described it in my coverage of that as a ‘shit sandwich’. They wanted to hear the opposite, and it was set up — the worst part about it that was it was set up as ‘We have a really sweet solution’, and that really, it was, like, electric, I mean, all of a sudden nobody was really thinking that’s what they were gonna say.

So the next day — fast forward twenty-four hours. I’m on the escalators in the Moscone Center; I’m coming down, and right behind me is Phil Schiller, y’know, senior vice president, right underneath Steve Jobs, the guy did the MacWorld keynote, y’know, a couple weeks ago — and I go, I’m gonna introduce myself. I turn around, and I said ‘Hey, Phil, I’m John Gruber.’. He goes ‘Hey, John! It’s so great to finally meet you!’ And he was so happy, he totally recognized me, and then the next thing out of his mouth is: ‘I’ve gotta disagree with you about that “shit sandwich” thing.’. And I… I mean, I was just like absolutely blown away that Phil Schiller, in the twenty-four hours after, like, a huge WWDC keynote, had gone to Daring Fireball and read my site. And then we had, we had a wonderful, it was absolutely phenomenal, like ten minute thing where we talked and, y’know, he told me how I was wrong, and y’know.

MM: Yeah, well, that’s…

JG: But it blew me away, it blew me away. Senior vice president at Apple read my thing about…

MM: That’s giant. For like, anybody, especially if you write stuff, there’s no greater thrill than having somebody that you know and admire go ‘I’m even aware you exist.’. But, like, for them to go ‘I enjoyed that thing you did’, like, somebody said hi outside a minute ago and ‘I like that one thing’, and, like, that made my day. That’s a connection. ’Cause, like, I did a thing, and I do this weird stuff that I can’t explain to my family, and I don’t understand how my daughter eats, but, like, somebody comes up, and goes ‘Hey, truck spank’, or goes, like, ‘Hey, Hipster PDA’, and I’m like, oh, man, that’s… Like, I’m not making this shit up, that’s awesome.

And if I’d gone out there and tried to figure out, like, how to be somebody else that was already that person, right? Like, I really like Cory Doctorow, but we already… There’s this great line, Ira Glass does this wonderful series of videos — Ira Glass from This American Life — and he has this great quote where he says, he says changes voice ‘The problem is a lot of people, they go out, and they wanna be’ — that’s my Ira Glass, or maybe it’s my Alex Bloomberg, but um — that’s funny to five people, but I’m glad you got it. That’s… see? duh. He goes ‘The thing is, people go out there, and they’re always trying to emulate the success of other people, right? And so you get on TV, and you try to pretend you’re Ted Koppel. But you know what? They’ve already got a Ted Koppel. They don’t need you.’ So y’know, like, your competition is somebody who had a unique opportunity a long time ago, and now you’re gonna try to, like, trace the shadow of that on a sidewalk and hope it’s a career? Right? It’s… we’ve got our Koppel, now who are you?

JG: And our instincts, I think, serve us wrong; and I, we call it’s like a lizard brain thing. But, like, our instincts tell us that if you want to write something — I mean, and that is part of these assumptions that we’re making, is that if you want to write, and we say write because that’s what we do, but again, it could be photography; it could be, y’know, a series, just making, like, a short film a week; any kind of thing. But obviously the whole reason you’re publishing it is that you do want to find a readership, or watchership — what do you call the people who watch videos? I dunno — listenership for a podcast; but you wanna find an audience. And I mean it’s, y’know, who knows? Who knows what the drive is for that.

But the irony is… the mismatch is that our instincts tell us that if you want to find an audience, you should try to find something that is like the things people are already enjoying.

MM: Right. Like how many of you guys — oh, you’re probably not old enough to remember this. But, like, after Star Wars came out, between like 01978 and 01980, there was, like, an unbelievable crap of movies and TV shows that were just unbelievably bad, because they wanted to cash in on the Star Wars thing. And yeah, they probably made a little bit of dough, but apart from Battlestar Galactica — and that’s the old one; don’t get mad, don’t write letters — but the, but, like, could you name a bunch, I’m sure some of you can; why am I even saying this.

nerd voice ‘So, actually, Space Wars ’79 featured Dan Blocker, who had been on Bonanza. He was the… so…’. But…

Ted Koppel was Ted Koppel because a bunch of Americans got kidnapped, and Americans cared a lot about what happened to them. And so they started this little show, for a half hour, every night. Right?

JG: Right. It started, Nightline started with the Iranian hostage situation. Forty-four Americans in Iran, and they weren’t gonna let ’em go.

MM: The country, like — you don’t remember this, ’cause you’re all, like, twenty, and with the SMS — but, like, America was gripped. This was a country that had not had its ass kicked in a pretty long time, and we were totally gobsmacked by what to do about ‘a bunch of people in the Middle East’ — that’s air quotes, if you’re listening in audio — what are we gonna do with these crazy people who took… America was gripped, right?

JG: Yeah, ’cause I guess it was like a sense of impotence because they’ve got them, and we’re like ‘Give ’em back.’, and they’re like ‘No.’. And that’s it.

MM: That’s it. Sorry.

JG: But, so, what did ABC News have? ABC News had the Peter Jennings show that’s on at 7 PM or 6:30 PM or whatever, and it’s a half hour, and they have to cover everything in the world; everything that goes on in Washington, and everything that goes on worldwide, and there’s a bit on sports, and… So, y’know, there’s, like, ninety seconds every night for an update on the Iranian hostage situation.

But it was this thing that people were obsessed about; Ted Koppel was completely obsessed about. And so he was like, ‘All right, why don’t we do a thirty minute show; what time is open? When can I get on?’ And they’re like, ‘Uh, 11:30 PM? 12 AM maybe twelve?’

MM: Against Johnny Carson?

JG: Right. ‘You get on… we got nothing.’ They were showing, like, the million dollar movie at the time. And so they just invented a totally new show: thirty minutes, every night, on the exact same topic every night, which is: everything new about this Iranian hostage situation.

MM: Right. And so, like, today, you go ‘Oh, you know what? I really admire Ted Koppel. I wanna be Ted Koppel.’ Well, you’re gonna need a couple things. The first one is you’re gonna need, like, a red wig, and then second you’re gonna need a time machine.

Because the reason Ted Koppel is Ted Koppel is not so different from the reason Michael Arrington is Michael Arrington or John Gruber is John Gruber. Which is, you cannot recreate the context, the timing, the everything of a moment where something happened, right?

I was saying this last night — who was I saying this to last night; Jim Coudal — I think, y’know, a lot of Americans don’t realize that in 01943, we didn’t know we were gonna win the Second World War. Right? If you’re twenty, you just assume that we always used to win wars. In 01943 — ask your grandparents — it was pretty freaky, man. People didn’t know what was gonna happen.

We lose that, when we just try to, let’s say for example; I dunno, take any example of a site that you admire. And instead of just focusing on the voice, or just focusing on the topic. There’s no way to recreate somebody else’s success, and why would you try? Which is not a way to say I’m… and again, I think you have stuff to say about this, but I would say it’s not, I’m not saying don’t go out and, like, learn from somebody else’s playbook; I am in fact saying go do that. But there’s that [37signals thing] not long ago that you linked to. It was, like, ‘What is it you’re copying?’, y’know? Are you copying the right thing, when you try to repurpose somebody’s theoretical success for what you wanna do?

JG: Right. And so, and… how many people here have heard of the 37signals? Yeah. So it’s, they’ve got the Basecamp, which is their project management app. And they invented Ruby on Rails to power it, and you get all this stuff for free. And then after they got successful, there’s all these other web apps that have come out — and maybe it has nothing to do with project management, it’s not that people have tried to rip it off the app — but they’ll make another app that does something else, but it looks like a 37signals app. It’s just got all these little, like, visual cues that are very very distinct, and it’s theirs, and it’s…

MM: You can move the rows around, and it’s a clean design; y’know, it’s…

JG: And it’s like they’re copying the wrong thing. It’s like, it’s almost like, you see a Honda Accord, and you decide, ‘Oh, that’s a nice car, I’ll make one like that’, and then you just sit there and look at it, and you just end up with, like, a papier-mâché car, and you’re just using wet tissue paper. There’s nothing to it; it’s just hollow. Whereas, the thing that’s worth copying is the attitude that they had at the outset; what made them do it. And project management meant things like Gantt charts, and…

MM: Every project — and I was a project manager — and everything that was out there — no offense against any of the apps, but — it was like, assumed that making software was necessarily like making a bridge, instead of being a little more agile, and just having the stuff you need, y’know. And their approach, it’s the same thing as Google. It would be like saying, ‘Oh, well, I’m gonna go create a white page with two buttons, and become the most successful company in the world.

Or, like, and my rant — and I’m sorry, I’m not gonna shut up about this — I’m so tired of every social media douche going ‘Zappo’s is on Twitter!’. And you’re like, ‘Yeah, they’re on Twitter; after putting millions of dollars into customer support.’ It’s like, getting an account on Twitter does not make you Zappo’s. Having the resources behind serving the shit out of your audience makes you Zappo’s. And it doesn’t happen overnight, with a login and an email that you click on a link. So anyway; not to go on a rant.

JG: Have you ever read… ’cause you know, Comcast is on Twitter.

MM: Comcast, they’re in Philadelphia, too, right? Comcast @responses very entertaining.

JG: They’ve built, like, a Death Star; they’ve, there’s a new; it is now the tallest skyscraper in Philadelphia. It is, like, it is, like, the tower of Hell that cable bills built. But if you wanna —

MM: The only difference is they’re destroying the planet more slowly than the Death Star.

JG: If you ever wanna entertain yourself on search.twitter.com, just type in @comcast, and read the things that people say to Comcast. Once it became known that Comcast was paying somebody to monitor Twitter for mentions of Comcast, it did not turn Comcast into Zappo’s.

MM: They should just hire Ryan King to just have a bot that writes back and goes ‘Sorry!’. Y’know? That’s really all they need. ‘@whoever just flamed me Sorry!’. ‘Your bandwidth? Your bandwidth has been, uh… Yeah, sorry!’. ‘Why can’t I get torrents?’

It is, it is… Y’know, I have this thing, again, everybody thinks I’m such a dick, because, like, I think social media’s more… I think it’s important enough to take seriously. I think that very much as to do with this. So people think ‘Oh, you’re so down on social media. Why are you such a jerk?’. And I’m like, ‘’Cause social media, when it’s really social media, is not about what you have to say; it’s having a tolerance for what people have to say about you.’ — which is so different from posting about your great run. Social media is when they say ‘You’re a jackass. Stop talking about your run.’ That’s social media. And that’s the conversation.

And I think kind of what we’re saying is, y’know, you do have to be open; there does have to be a certain amount of tolerance that you have for every aspect of this. The biggest tolerance that you’ve gotta have — and I’m as thin-skinned as anybody; I don’t like people saying mean things about me — but, I think what we’re saying, in some ways, is, you need a tolerance — this is gonna sound so unhelpful — you need a tolerance for having no idea where your thing is going. Y’knoww? ’Cause if you have too much of an idea of what it is, like, you may be accidentally making the wrong thing. If you’re not responding to what’s really happening; if you’re just going, like, ‘My goal is this. I’m going to have this thing, and I will have this many followers, and there will be this many comments, and I will have a rich community.’ And instead of going and listening to what people say, and making the thing, and…

It’s a real tightrope walk, because yeah, you do have to be arrogant enough to think that it matters to try at this stuff, and yeah, you do have to be arrogant enough to look at stats and see what kind of material people enjoy, but there’s all kinds of ways… we were talking about this earlier; I was like, ‘All this social media stuff is like a giant set of extremely sharp knives, where, like, they’re just knives, but you can use them for good or ill.’ Like, SEO? SEO’s fantastic, because it gives people URLs that make sense. But it does, y’know… it is bad if you’re trying to fool people into clicking things.

But… I dunno, we should probably move on. How we doing on time? Oh, we’re doing great on time! We should slow it down, that’s terrific.

JG: I think that the big irony is that there’s this old maxim, I dunno, it’s probably… I wish there were some kind of, like, thing where you could just…

MM: Search the web?

JG: Right. That would… I tried to look for —

MM: nerd voice ‘So, what was that reference that you made in your earlier tweet? What was that?’

JG: So it’s probably, I should probably know who to attribute it to, but maybe it’s not; maybe it’s something that’s been around forever. But there’s a saying that it’s great that we have freedom of the press in this country, but the only people who really have freedom of the press are those who can afford a printing press. And it’s totally true. I mean, it was, y’know. You could not reach — you could say what you wanted, but you could not reach a big crowd unless you had the money to reach them — and a television station costs gazillions of dollars, and printing newspapers, even in the old days, newspapers…

MM: Even in the eighties, just making a zine, just going to Kinko’s and making a zine, and having to, like, pay postage on that? It was extraordinary. And you’d still reach, like, a tiny fraction of people. Like, Maximum R&R, like, what was Maximum R&R’s greatest circulation? Like, y’know, going on newsprint… It’s like, today, everybody owns a little press.

JG: Or look at Boing Boing. It was a zine, and I never even heard of it. But I mean, it was apparently very popular. But —

MM: It was about ukuleles, I think.

JG: Right, something like that. But, I mean, but then the Internet, it literally is the solution to that problem, where everybody can afford their own printing press, and can reach tremendous scale. I mean, a seven-dollar–a–month web hosting account will almost certainly saturate… you will be able to satisfy anybody who could even be vaguely interested in what you will say. It’s unbelievable. I mean, you could go to Tumblr, and Tumblr, what… it’s free, right?

MM: It’s free. Marco Arment’s here. Shoutout for Marco. Anybody like Tumblr? Anybody like Tumblr? Marco’s here, give him a big hand.

No, it’s totally true. And what’s funny is I started doing this stuff related to this web stuff in the mid-’90s, and I had to hand… I sat there with BBEdit, and Fetch, and had to, like, go… I mean, someday I’m gonna tell you guys about when I ran a giant conference site by outputting flat files out of Filemaker Pro with a script. That’s how you used to publish, if you were me and you didn’t understand Perl and Apache. Like, you had GoLive CyberStudio, y’know?

But, like, so, I think it’s interesting; I still remember, like, the Peter Merholz

s of the world, like, having conversations about, well — or Rebecca Blood talking about ‘Is this a blog? Or is that a blog? Like, what’s a blog?’ And I think now, I’m not even sure ‘blog’ is that great of a term anymore, to describe anything, ’cause it can mean, it could mean Gawker, or it could mean something on Blogspot, or it could mean, y’know, some incredibly awful corporate site that’s basically press releases with a permalink. And that’s really different from me going and hand-coding every page. So… it’s important to acknowledge that, like you say, it’s not that we want for tools; if anything — God, people hate me — I mean, one thing that bugs me is, like, if I don’t post on Twitter, I get 7% more followers per day. Because I’m annoying.

JG: I think…

MM: Because words are harder than buttons. Y’knoww? That’s the problem. It’s super-easy to post nowadays…

JG: I think what’s funnier is that you know… you’ve actually looked and studied the statistics.

MM: No, I ran it in Numbers, I ran it in Numbers, ’cause I stopped tooting for several weeks, and it kept going up, like inexplicably. It’s like, what are you following? I’m not here! It’s like, and it wasn’t that funny to begin with. It’s just like…

But I guess what I’m trying to say is… and I’m not trying to play the douche card, and say, like, you’ve gotta be any way. What I’m saying is that the tolerance that I’m encouraging you to have is, first of all, a tolerance that, if there’s something that you’re kind of into doing, that you’re pretty excited about, and think about a lot, y’know…

Oh, so what was it we were talking about earlier? Like, how do you know that you should probably start a blog? Like, people keep telling you to shut up. Right? You’re like, ‘Oh, whatever, Cowboys! I love the Cowboys! nonsense sound The Cowboys!’ Like, y’know what? If you love the Cowboys, like, why don’t you either gay-marry them, or start a blog. Right?

JG: But that’s…

MM: And how do you know? Do you go, ‘Oh, what’s a popular topic? Web 2.0.’. Or do you go, like, ‘I really…’, like, look at, like Perez Hilton. Like, I don’t love Perez Hilton’s site, but you so know Perez Hilton. I’m not a giant TechCrunch reader, but you totally know TechCrunch when you see it, y’know? It’s, like, they’re obsessed with certain things.

JG: Right. I mean… with TechCrunch, and it’s… that’s one of those sites that because it has become so popular, and people talk about it being worth, y’know, $20 million or whatever. So then all the people who, going all the way back to the beginning, and who start with the idea of, ‘Okay, I wanna make a lot of dough on the Internet, with a website, so who should I copy?’. And then they look at TechCrunch, and then they copy the format, they copy the things that he writes about, but the way TechCrunch started was that Mike Arrington who, I agree, he’s a total dick.

MM: I didn’t say that. I did not say that. Hey, easy, easy!

JG: I didn’t —

MM: He’s got parents, be nice.

JG: I didn’t mean it.

MM: You don’t think he’s a dick?

JG: Oh, he’s a…

MM: He’s got some journalism kind of…

JG: No, I meant it, I meant it.

MM: He seems okay. He seems…

JG: No, he’s a dick. He’s a total dick.

MM: Can I point out one thing in passing? A bunch of you — I’m sure people are gonna go toot about this now, or whatever, and John’s gonna have to go get in a fight — but can I just point out why I love John Gruber? One of the reasons is John Gruber so doesn’t care if you agree with him. Right? And like, yeah, whatever, the two of us are dicks, but like, y’know? I so admire people who don’t need me to love them. I have so much affection for somebody who really believes something and their belief and interest in something is way more important than me pretending to like them. I just have so much admiration for that. And to the extent that you can, and in the way that you need to for what you do, I think you have to do that too. You don’t have to be a jerk about it, but I think figuring out, it’s, like, okay to have a strong voice about something. Right?

Look at, like, Rush Limbaugh and Ann Coulter. I am not persuaded that that many people agree 100% with Rush Limbaugh and Ann Coulter. But y’know what you’re getting. It’s like, it’s like watching pro wrestling, right? It’s like, they’re characters, and they have a voice. And personally I don’t find them very, like… I don’t find what they have to say very useful, but I get why people listen to them — or Howard Stern. I’m getting a little off-topic.

But I guess what I’m saying is — we should move on to this next bit — the reason, where I’m going with that tolerance is just this idea of — we should get into the money part. Yeah?

JG: We should. Just one more thing before we go to the money part, is with the Arrington thing, is that Arrington…

MM: nerd voice ‘So, @everyone? @gruber says @techcrunch is @dick.’

JG: It fits right in with our general thing where you find your obsession —

MM:RT @gruber’…

JG: You find your obsession that no one else is writing about, and then you just pour yourself into it. And then here’s a guy who for, whatever reason, his obsession is venture capital funding for web startups in the San Francisco Bay Area. Which is a dirty, rotten, disgusting business, and it’s just vile. And so no wonder the site is dirty, rotten, disgusting, and vile — it’s a rotten, disgusting topic. But that’s why —

MM: He’s a lawyer. He’s a lawyer. Like, he knows what forms to fill out to ruin you.

JG: Right, and that…

MM: Like, he’s probably got, he’s probably got like three interns that do nothing but fill out forms to ruin people all day long.

JG: No, I —

MM: He’s got chunks of guys like you in his poop.

JG: I know that that’s why, I know that he’s a lawyer, and I know that that’s why he does that bullshit thing…

MM: He’s mad, like, he’ll… he’ll punch a bitch, I’m pretty sure.

JG: I’m pretty sure I’m faster than Mike Arrington.

MM: You think so? He seems he might be a little logie.

JG: I saw, I met him once, and I kinda, like, sized him up. He’s kinda doughy.

MM: I thought he… I thought, I thought he pretty nice. Like, and we’ve met a couple times… no, I’m being straight up. I don’t enjoy his site that much but I like, he seems like a nice enough guy.

Anyway, what I was gonna say was — ’cause I want, I don’t want, really, I love John’s wife and his son a lot and I think we should move on — is that… that the final, for the end of act two, the final ambiguity that I would like you to think about is ambiguity about how this turns into a way for you to become rich on the Internet.

Because it’s… contrary to what a lot of social media and blogging douches will tell you, it’s not easy. And a lot of people who act like they’re making an assload of money are just full of crap. It’s really… I mean, I’m not saying it’s a hard job, but I am saying do not assume that everybody who has ads on their site is making a killing, regardless of what they say the CPM is.

Because the real opportunities of this stuff — this sounds like bullshit, but I am dead serious — the giant opportunities in this are not short-term gains… I’m giving you an opinion here, which I don’t usually do. But the real long-term gains for you are not pageviews and CPMs; it’s the opportunities that come out of being awesome at what you do. And if you think that’s BS then, like, I can’t help you.

But I swear to God, if you look at the people around who seem like they were born on third base, yeah, it’s good timing; yeah, it’s hard work; but I think a lot of it is they had a tolerance for the ambiguity about where it was gonna go, they had a tolerance for the fact they were not gonna take short-term money that got in the way of what they really wanted to do. And the ancillary revenue streams and opportunities that come up as a result of making extremely-high-quality content…

I mean, has there ever been a better time to make something awesome on the Internet? People don’t have money to buy things anymore. I don’t know if you know this; there’s no money. There’s… if you’re lucky they’re on dialup connections; maybe they’re at Barnes & Noble; somebody wants to look at a computer at the Apple store and they wanna check their Facebook… People don’t have money. And they’re looking; they’re starved for content that speaks to them, that’s not a reality show. I’m gonna stop ranting, but I think that’s important.

JG: But there’s… it’s like those Mastercard commercials, where there’s more than money can buy. And it’s, oh, terribly trite, and so obvious, and not interesting, and we’re all bright, clever people, and so we don’t really think about little canards that aren’t very interesting. But a lot of times they’re very true, they’re totally true; and there are things that money cannot buy that have tremendous value.

And one of them — I mean, you’re Merlin practically making a career on it — is that attention, human attention, is valuable and it is limited. There is nothing you can possibly do give one person more attention in a day. You wake up; you have eighteen hours; and then you go to sleep. And in that time, you only have so much attention. It’s a limited resource. You can’t directly buy it. You can’t… there’s no dollar value on it.

MM: Right. And it accretes over time.

JG: But it is incredibly valuable. And so that is the one thing that when you give stuff away in the Internet, it’s like, well then how am I gonna get paid for it? Well, you’re gonna get paid in attention. And I know you cannot pay your rent, I mean, I know…

Continued in pt. 2

written 27 March, 02009 Comments

Transcript and commentary for ‘Whither Magnolia’

Transcription of Citizen Garden episode 11, ‘Whither Magnolia’.

As you may or may not be aware, the Ma.gnolia bookmarking service recently lost its entire database.

I was not personally affected by this loss, as I instead use delicious and back up my bookmarks daily. I had briefly tried Ma.gnolia; after a lengthy wait while it processed my bookmark collection, I soon decided that the system simply didn’t allow for the things I wanted to do.

But even if I didn’t lose anything, a lot of people did. And if delicious were to disappear, I might still have my data, but I wouldn’t have a way to use it (immediately, anyhow; there are open-source importers). So this is still a good thing to think about.

Although Chris Messina doesn’t seem overly concerned with the loss of his bookmarks, I make heavy use of my bookmark archive — collecting things like article series, free-culture content, free music, and a variety of other purposes. For me, it doesn’t have a half-life of twenty-four hours; I bookmark so that I can quickly re-find things that have interested me. I’m willing to grant that I’m an outlier; the number of tags I use per bookmark likely ensures that anyways.

But even if bookmarking is done on a very short-term basis, it’s useful, as the podcast points out, for things like generating recommendations. A major trend in my feed subscription habits is that I love sites which point out things I’d never see otherwise. As Dave Winer says, People come back to places that send them away. Although I didn’t use Ma.gnolia in its pre-crash form, I’d be very interested in one that tried to give me a list of interesting links. I’ve lately begun skimming the front pages of Digg and Reddit several times a day — which, while useful, also means I have to see a lot of things that I really don’t care about. Links recommended by a computer aren’t quite on the level of links recommended by people I trust to be interesting, but it can be very close.

Regardless of whether Ma.gnolia ever appeals to me personally, I hope it comes back stronger than it was. Competition is good, and the service has a chance to move things in interesting new directions.

On to the transcript.

Participants

Transcript

LH: Hello, and welcome to episode eleven of the Citizen Garden podcast. We’re actually coming to you today via video as well. I’m Larry Halff.

CM: I’m Chris Messina.

LH: And today we are going to talk about what happened with Ma.gnolia.

CM: Yeah. Which I guess is, for many people, not that funny, but uh, it’s fairly, I guess, sort of a momentous thing, and of course you being the news creates an interesting opportunity for us, I guess, to both talk about what happened, and for you to sort of explain maybe the situation as it occurred, what’s happened since, what you’ve done sort of in response, and maybe some lessons learned here. So maybe you wanna give us some background on what actually happened.

LH: Uh, so, I still don’t have all the details on what happened; still working with a? data recovery company and hope to know more when I hear back from them, but: what seemed to me to have happened was we had some file system corruption and our very large database file got corrupt and…

CM: What size database file are we talking here?

LH: It was approaching half a terabyte of everything together, and…

CM: Is this MySQL, or…

LH: Yeah, MySQL; MySQL 5.

CM: Okay.

LH: And… I think this had been somewhat of an ongoing issue, but everything was running even though this was going on. And eventually it stopped running, and the site went down. It just no longer worked. And because of this, our not-so-awesome backup system also failed, because it was not able to properly back up the data from MySQL.

CM: Is that because of the size, or I mean… so what, maybe you can talk a little bit about what you know what happened with the backup.

LH: So what happened with the backup was it was just trying to back up bad data, so whatever the backup produced was not usable either. It was just giving a file sync over a Firewire network to a different machine. So, in this case, because we didn’t have a good sort of integrity-oriented backup system, it failed.

CM: Now, had you ever done tests or anything like that to see..?

LH: Nope. Had not purposely failed the database to see what would happen.

CM: I see, so…

LH: Which would… which is one of those lessons learned, which is: test your backups, test your backup system. I don’t know that a test would have caught this sort of thing, but it’s something we should have done. And another lesson learned would be: figure out your backup, figure out a good versioned backup system early on. Or actually the real lesson learned is if you’re a startup, don’t do your own IT at all, which is… And I think three years ago, it was less of an eff — three, four years ago it was less of an option. Ma.gnolia, I really started on Ma.gnolia four years ago, and we were running the beta over three years ago. And there was not… there was no sort of cloud computing at that time. So it was the, you know, the option was really bad hosting, especially for Rails applications, hosting that…

CM: It almost didn’t exist back then.

LH: I knew wouldn’t scale. Or do-it-yourself. And sort of in the process of developing Ma.gnolia, infrastructure always sort of took a back seat. And along the way we suffered because of that; y’know, I’d say in about 02006 we definitely attrition from the service because of speed and reliability.

CM: Performance.

LH: Yeah, the performance, the site would slow down. But because you… because we were developing Ma.gnolia specifically for the environment it was deployed in, there was… there is a huge tax to sort of moving to a completely new environment. We have all sorts of dependencies, all sorts of stuff that we required in our specific environment.

CM: Now, I mean, maybe you could talk a little about the actual infrastructure, y’know, the environment you had set up, from a hardware perspective. Because I think one of the things that, y’know, most people probably have no insight into, y’know — unlike your Mac you can’t go to the little Apple, y’know, and choose ‘tell me about this Mac’, and get the specs.

LH: Right.

CM: Y’know, for web apps. And maybe you can talk about, you know, the actual system that you were running Ma.gnolia off of.

LH: So we were running Ma.gnolia on… the database and the backup were on a couple of Xserves, and then we had about four minis

CM: Mac minis.

LH: Mac minis. ? Mac minis that were running as frontend web servers. So it was a very small setup, and… I mean, interestingly, y’know, with a pretty good Xserve as the main database server, it ran pretty well.

CM: And you weren’t doing anything like RAID or anything with it, it was just Firewire backup.

LH: The server was RAID. Its disk was RAID, so that’s one of the things we’re looking at. But it was a software RAID, so if it’s a filesystem problem then… that’s not gonna do any good because the the errors were RAIDed as well.

CM: So let’s talk a little bit about, I mean, the reaction, y’know, to this so far. The reaction I’ve seen has been somewhat mixed. Y’know, there obviously was sic some articles that came out, that sorta like, rightly pointed out that this was a bad thing that happened, and yet — I guess maybe you can speak to, because obviously you’re directly involved with it — the reaction from both individual users of Ma.gnolia, y’know, as well as, y’know, sorta like the larger media that’s like Wired and so on.

LH: So I think, um, the reaction has been actually mostly supportive and understanding.

CM: Yeah, I’ve seen a lot of that.

LH: I’d say ninety percent of what I’ve been getting and reading has been, y’know, not tearing down, not flaming, not griefing. And the negative reactions out there, I think a lot of them are valid. It’s… I made a huge mistake in terms of how I set up our system, and the — when people criticize that, they’re completely right. I have no problem with that. There have been some personal attacks, but… I think people get frustrated, rightly frustrated, and angry and sort of fall back into that mode, where they want someone to go after, and make it personal since they feel like they were personally let down.

CM: It’s also been interesting to see the characterizations of ‘Ma.gnolia and co.’, or ‘Ma.gnolia and team’, as though you’re this large operation, y’know, with international offices and things like that. I mean…

LH: I think that’s another lesson learned, which is, like, we always appeared bigger than we were. And it was me, and it’s basically been me. As of late, there was… for much of Ma.gnolia’s life, there was a small team; I think the largest we ever got was four. So we somehow projected this image of this, you know, big company with, you know, huge offices and cubicles and the whole works, and it was, you know… it’s really just, it’s really just basically me. And I don’t think… I mean, I think it was flattering that that’s the perception, but I think it was a mistake to not work harder to let people know exactly what we were and how big we were, in terms of how personal the service actually is.

CM: That says, I think that says a lot to a lot of the lessons coming out of social media lately; I think, y’know, around transparency and openness, which, obviously, Obama says a lot about, but there is some degree of truth there. Now…

LH: I don’t think it’s something we ever hid.

CM: Right.

LH: In fact, I was always surprised when… in terms of how large people thought they were. In fact, I was surprised at how much news coverage this whole event got, because Ma.gnolia is very small, even in terms of its user base, it’s very, very small. It’s just com… insignificant compared to any of the other real-web applications out there. But it somehow always projected this image of being this much bigger thing than it actually was.

CM: Yeah, I mean, even though it’s a small team, or just you, most of the time, I don’t think that that necessarily excuses what happened, but helps, maybe, to put in perspective, y’know, both from this hardware perspective, y’know, mostly you keeping it up and running, mostly you doing a lot of the work on these things that… I guess there are two things that can come out of this. One is that an individual can actually build a fairly, y’know, substantial community, relatively speaking, with the tools that exist today… that a lot of these tools are more accessible than maybe they once were. For example, you mentioned that the commodity-hosting thing sort of, y’know, that was the way that you could do it, which isn’t great; or you could do it yourself, and bear those possible risks and consequences. But it also says, I mean — this is, maybe, y’know, this is an opportunity to go back to where Ma.gnolia came from. I mean, I found Ma.gnolia a long time ago largely because I read the web standards book, by the ‘blue beanie guy’.

LH: Jeffrey Zeldman.

CM: Exactly. And he, of course, Happy Cog did the design of Ma.gnolia originally, and that’s how I originally found it. And Jason Santa Maria did the logo, and I was like, wow, this is a great-looking site, I really wanna use this, y’know, it looks kind of interesting. And yet I had no real insight into where it had come from. I mean, maybe you can talk about the germination of the site, the work that came before, that led to Ma.gnolia, and y’know… what maybe your goals were originally.

LH: Well, it’s been a sort of long and shifting road, but to go way way back, my background is in cultural anthropology, and I did my graduate work developing qualitative research tools. And sort of… I think I sort of revisit that every so often, and Ma.gnolia’s one of the revisitations of that work, and in a way it’s a tool that helps people gather disparate information, and thread it together in ways that make sense to them and their community. So, that’s sort of like the way-back origin of Ma.gnolia.

CM: Well what were your goals back then?

LH: So I think… I mean, it’s funny, I think my goals when I started Ma.gnolia were were to — at that point, you could make a lot more money on site advertising, so the idea was to build, was much more around a publishing model, and… sort of, as we launched the site, and as we watched people starting to use it, it was clear that that was not what this was going to be. And also was not necessarily true to my background and my work and who I was. And so, as… throughout the beta period, and the initial months, and the launch, we pretty quickly refocused the site on collaborative, community, developing-type tool, rather than just a publishing/advertising-type site. And in fact, ads were designed in the original site, and I never turned them on until… well, I think actually they were on briefly, but basically I left ads off the site until I added the ads-off upgrade. So that’s sort of the initial start. And going down that road definitely was the right thing of the site. I think it really found its identity, and really had a vision and message as a community site.

CM: So would you say that overall, you know, notwithstanding what happened recently, the site was a success?

LH: I think the site was a success in terms of what it brought to people’s lives, the community it developed… it definitely was like attracting like, in terms of like attracted people who cared about their… environment, I think, capital-e Environment, in terms of, like, not just the way the site looked and the way the site acted and the interaction, but also, like, the people around them, and the other people on the site. It attracted people who were thoughtful and caring, I think. So, yeah, I mean, in that sense it was a great success, in terms of, I was able to build something I loved for people I liked and respected. The site didn’t ever actually make money, was not a business or financial success.

CM: So from that perspective, you essentially were bankrolling the project, kind of maybe out of a hope at some point it might turn into some sort of business or something, but for the most part it sounds like it was a labor of love, which a large number of people eventually ended up sort of relying on and using on a fairly regular basis.

LH: Yes. This was definitely a labor of love. I was doing this because I loved to do it. And it was completely self-funded. I would have loved to, and I was working towards building a business model around it, with the add-ons, and I was working towards, y’know, bringing ads back into the site in a way that was more relevant. But some of, a lot of those plans never got executed on.

CM: Well, so let’s talk about that. I mean, there are a couple things that have changed in the last several years, largely, many more people are using social networks now, and there’s a lot more people online. The competitive marketplace for sharing bookmarks is probably heated up a bit, even though delicious is probably still the heavyweight, y’know, in the area, just because of Yahoo!’s involvement…

LH: I actually think, I mean, I think Ma.gnolia was a unique thing in terms of the community that organized around it. But I think that, in a way, I don’t think we could ever compete in the real world of link-sharing. I think the biggest link-sharing site right now is Facebook. The people are sharing their stuff: photos, links, any of the stuff in context of the communities they’re already hanging out in.

CM: Sure.

LH: So the destination of social bookmarking, I’m not sure where that’s going.

CM: So let’s talk about that, then. I mean, obviously, there’s sorta been a quiet lull after the storm, if you will, where I think, you know, you need an opportunity to maybe collect your thoughts about what happens next. But what are you sort of leaning towards right now? I mean, not all the data has been retrieved yet, or recovered yet; there are a number of tools that you’ve made available, which you probably could talk a little bit about, but in any case, whether the data is able to be retrieved, and people are able to get their bookmarks out of Ma.gnolia, is the question of, well, what happens in the future? Y’know, three, six, nine months from now, has Ma.gnolia recovered, has it come back? Because I think if you make that distinction between the data and the community, there’s something there. There’s a social fabric that was created that, though, ripped out of a context because the social objects went away, there’s still people who probably would like to continue connecting and sharing with one another. So…

LH: Yeah, and the community has asked for the tool back.

CM: Yeah, they want the tool.

LH: So that’ll be coming back, in a modified, in a different sort of way. It’ll be coming back in a proper hosting environment, for one thing.

CM: Where you’re not responsible for IT any more?

LH: Yeah. It is gonna go into a more reliable utility…

CM: Where Werner Vogels is responsible, the guy over at Amazon?

LH: Exactly. And with better backup safeguards in place. I think that’s my first priority in bringing it back, is… I mean, you could never guarantee anyone a hundred percent of anything, but I can get a lot closer than I was in the prior setup. So that’s sort of the biggest change that’s gonna happen, in terms of technically how the site is gonna change. In terms of how the community is gonna change, it’s gonna, it’s going to… it’s sort of, I think as like going back into private beta, that I’m not going to have it open registration, that the site is going to relaunch by invitation only, and then slowly build up from there. And definitely people who were part of Ma.gnolia 1, who were good community members there, will be invited back to join from the outset.

CM: Now do you think, I mean, that people can trust you again? Or do you think this is just gonna be something that you earn back over time?

LH: I think it’s gonna be something I earn back over time. I’m gonna completely disclose what our infrastructure is, when that’s settled on, and let people make the call based on that.

CM: Yeah, I think, y’know, it’s sort of raised a number of questions, I guess, in my mind, about… y’know, again, without that kind of, y’know, Apple menu, you see what’s behind these services. I use a lot of web services, and I had about 6300 bookmarks on Ma.gnolia, but I have similar sort of quantities of stuff — data capital, as I call it — strewn throughout the web on other services, for which I have no concept or idea of how they perform backups. So it’s been interesting for me to, y’know, witness some folks in the Get Satisfaction forums were coming and, y’know, making these claims about oh, this is preventable, and you could’ve done this or done that, and sort of, y’know, playing armchair IT professional and saying, well you could’ve done all these different things, but at the same time we don’t have a great deal of disclosure from other web services too. So it’s not just that Ma.gnolia was the only one doing this, it’s that there was an experience here that sort of sheds some light on these different IT practices, I guess, for better or worse. I imagine that there are a lot of other, for example, applications out there, y’know, that are built to serve the Twitter community, that are probably equally, if not much worse off, than Ma.gnolia from an infrastructure perspective. So it raises a broader question, I think, about, y’know, who we are entrusting with our data, and where we’re putting it, where we’re hosting it… And in some ways, making sure that there is a personal sort of connection or relationship there, I think it becomes more important over time. I mean, if you imagine these services as kind of like your bank, and you wanna entrust your bank, y’know, over time I think that individuals, now that you’ve had this experience, you’re never gonna repeat this problem, y’know, this situation. Other services may have to similarly have that kind of experience until we realize this is actually a big deal, and this is a long-term sort of, y’know, consideration to make. I mean, a lot of the work that, let’s say, I do with OpenID is around also helping OpenID providers understand and realize the gravity of their purpose. Y’know, it’s just like if your email went away, what would you do? For a lot of people, I think that would be very very bad. So there’s that, is that question, like, sure, people could keep countless backups of their own data on their own machines, and things like that; and only recently, though with tools like Time Machine has even personal backups become somewhat more accessible. So this is, I think, a question for many more people than just either Ma.gnolia or the Ma.gnolia community or you. Where? it’s a question of, how do we go about making smarter decisions about where we host our data? And just because we can keep everything, what is the real value? And I think it’s yet to be seen; I mean, you talk about sort of the qualitative… what was it, the research that you said?

LH: ? Collaborative qualitative research tools.

CM: Yeah, so you can imagine that this… I have this sort of abstract concept of this, since I don’t really have to get too much into the bits and bytes. I don’t really need to think about how hard it would be to do this the right way, but, y’know, 6300 bookmarks gives you kind of a fingerprint of the stuff that I’ve consumed over the last, y’know, three or four years. You could imagine using that as a filter for things that might interest me in the future. And so, on the one hand, just having your bookmarks some place, to me, is not all that interesting. I have backups, y’know, from ages ago, and I have stuff I did in college on hard drives somewhere. I’m never gonna look at that stuff again, but I have this sort of abstract through that, oh, some day I’ll break out the old, y’know, 180 gig hard drive, or actually at this point probably 180 megabyte hard drive; I’ll be like, oh, take a look at that! Y’know, like, I did that with Photoshop 3. But… there’s just so much data now that you almost need to be living much more in the moment, doing these things in real time, where a bookmark has a half-life of, y’know, twenty-four hours, if not less.

LH: Yeah. I mean, I think you’re right. I mean, the interesting value in data like bookmarks is probably more along the lines of what… I mean, there is stuff you wanna go back and find, but a lot of the value is probably in terms of using that to build other tools, like what Apple did with the iTunes Genius —

CM: That’s right.

LH: — where it’s like, they can look at your entire music collection —

CM: And your listening habits.

LH: — and your listening habits and stuff… you may not be listening to, right now, an album you got five years ago, but it can bring that back, or use that to find other songs, in terms of developing those Genius playlists.

CM: I mean, y’know, if you think about it from that perspective, Ma.gnolia has — or had, and may still have — a great opportunity to start doing that, where it could be, y’know, your daily list of links, recommended to you based on your previous history. And that’s something I haven’t seen done a great deal; it’s very hard to do, very hard to get right.

LH: It’s computationally intensive.

CM: That’s right. But nonetheless, you can imagine that moving forward, that would be a very valuable way of making use of, y’know, this type of service. So, well anyways, maybe you… any parting thoughts? Like, y’know, to, let’s say, Ma.gnolia users, y’know, who are sort of waiting for something. Y’know, the next thing, like…

LH: Um… I mean, I just can’t thank people enough for their support. It’s really… as difficult as this experience has been for me, I think my faith in humanity has been reaffirmed…

CM: That’s good.

LH: And really, I have everybody out there who was hurt by this experience to thank for that. And just, also, keep an eye out for the updates on Twitter and Get Satisfaction and the Ma.gnolia homepage for in terms? of bringing the community back, I’d say in the next month, month and a half.

CM: That’s, that’s exciting. I mean, I’m looking forward to it regardless. Y’know, whether or not my bookmarks are there or not is actually not what I’m most interested in. I think that having it there, it’s one of those things where, you don’t really miss it until it’s gone, right. So now that we’ve had that sense, you wanna fill that void, and I think it’s good to know that, y’know, Ma.gnolia will be, will grow again.

LH: It will.

CM: All right, well, appreciate it.

LH: Thank you.

written 16 February, 02009 Comments

Transcript and commentary for ‘Opening Preconditions’

Transcription of Citizen Garden episode 9, ‘Opening Preconditions’.

This one was informative for me because although I was vaguely aware of Ma.gnolia’s plans for their version two release, I didn’t know about the technical aspects — requiring OAuth, self-hosting, and open source code.

The discussion about using OpenID and related ‘open web’ technologies to automatically tell web services where things is quite appealing. This podcast episode is from late August, so maybe work has already begun on implementing such concepts.

But the most interesting part, to my mind, was the idea of using the fact that services tend to come in categories — bookmarks, social networks, and so on — we can export data from those services on a regular schedule and back it up for safekeeping. Then, if we decide to move services (e.g. MySpace to Facebook) or the service goes offline, we’ll have full access and control over our ‘social objects’.

If you think about it, this is really why the open web movement exists. The open web revolves around the idea that if I put information into the system then that data belongs to me. If I fill out a Facebook profile or use Twitter or make a bookmark on delicious, it’s true that somebody owns the system that allows those things — but what use would the system be if I didn’t use it?

So it’s about ownership. But ownership implies freedom — freedom to do whatever I want with my information, since I control who has it and what they can do with it.

This is something that companies offering hosted services dislike. Their business model is generally based on the idea that once you put your information into the system it’s stuck there. Investors see much more revenue potential in a captive audience, because the audience has no choice but to use whatever’s being offered, unimpressive as it may be — a non-technical example would be product placement.

And this is where the whole idea really becomes useful. An important part of the open web is data portability — the ability to easily transfer my information between services. Suppose I’m on Facebook, and I decide I want to try a different social network. With data portability, I’m able to tell the new site about my existing Facebook account and let it import all my friends.

This means that the power situation is changed. I’m no longer stuck using whatever site my friends have chosen. I’m no longer forced to trust that the site I’m on will continue to provide new features and a useful service. Instead, by using a service I’m actually endorsing it, because I’m not obligated to stay there.

It also entices the site’s owner to improve their service. When a site maintains its user base through feature offerings, they’re less able to simply make something good and then let it sit. Innovation is hard, but when it works everybody wins.

But despite all that, I set out to publish a transcript, not an essay. Here we go!

Participants

Transcript

LH: Hello!, and welcome to episode nine of Citizen Garden. I’m Larry Halff…

CM: I’m Chris Messina.

SK: I’m Scott Kveton.

WN: And I’m Will Norris.

LH: And we’re here today… I’m freshly back from Seattle after making a big announcement at Gnomedex that Ma.gnolia is being rewritten from the ground up as an open source, downloadable tool. Actually, it’s being broken up into several pieces. I think that’s the more important announcement; there’s a lot of open source publishing platforms out there… blogging tools, there’s even, y’know, I think, a handful of open source social bookmarking systems. A more important thing about what we’re doing is that we’re really trying to pave the way forward with the open web, and part of that is getting these decentralized and federated systems talking to each other. And open source happens to be one way to advance that cause.

CM: Let’s back up here a little bit, ‘cause I’d like to sort of understand better where this idea came from — why you would take this approach when, y’know, you’ve built up a pretty good audience on Ma.gnolia, you’ve got a good user base there, and you’ve done really well so far in supporting a number of these open protocols like OpenID and supporting microformats and OAuth and so forth. but how does open source or open sourcing the platform actually support the open web — and what actually motivated that decision?

LH: Well, I think the primary motivation was seeing that the social aspects of these publishing tools doesn’t really scale when there’s the big single point of failure. and Ma.gnolia has had downtime, Twitter has the infamous ‘fail whale’, Flickr gets massages — and all these things happen, and when it happens you sort of… you lose access to a pretty important piece the flow of your online life. And as they grow, the load that’s caused by exponentially putting out… the exponential effect of putting out all these social connections and publishing and keeping everyone up-to-date just doesn’t… just isn’t really gonna scale in the long term. And I think, y’know, what we’re seeing with Twitter now is nothing like what we’re gonna see with that kind of tool down the road. I mean, hardly anyone really uses Twitter…

SK: Yeah, like a million users maybe.

LH: So I think we really see an important next step is to finding out a way that these things can be pushed out to the edges, yet still have the social functionality of getting everyone talking to one another.

CM: Now one of the things I think, y’know is interesting… so, Scott is with us at Vidoop; he’s also the chair of the OpenID Foundation, and he’s, y’know, sort of one of the champions of decentralizing identity and things like that. I think it’s sort of interesting to think about how OpenID in some ways creates the preconditions for Ma.gnolia going open source, by allowing people essentially to have sort of a cargo horse on which they stack a bunch of things — their photos, their bookmarks, and so forth. Maybe you can — Scott — talk to us a little bit about how you see OpenID maybe as that beginning point that… allows for this type of decentralization that Larry’s talking about, where there are much fewer single points of failures, or at least these concentrations that maybe are creating these pressures on the network.

SK: Okay! Uh, yeah, absolutely! So there’s, y’know, we’re three years into OpenID now, and it was really funny at OSCON, actually — and I think Chris you were on that panel with me, weren’t you?

CM: Possibly, yeah.

SK: Leah Culver got up…

CM: Oh, right.

SK: … started just dropping f-bombs about how she just thought OpenID was dumb, she’d never use it, da da da da. And y’know, in reality, it’s… it is a URL-based system, and that’s just… users have not, y’know, grokked that. But I think what we’re starting to realize is the value is proving a potential service endpoint, which means nothing to users.

CM: [Nor necessarily?] should it…

SK: Right, absolutely.

CM: For now.

SK: … but for developers it’s extremely important. So, y’know, some of the challenges that we’ve been facing in the OpenID community are around security and usability. If we can make it easier for people to identify themselves — whether with an email address or identity in the browser — then they can get in, prove that they are some end point without having to know what that endpoint is, and then start to put their data there. and then we start that… that lays the groundwork for things like y’know, lower-case data portability and the ability to, y’know, have more control over who provides your… or manages your data.

CM: Yeah, I’ve been thinking about these things lately, and it… it’s interesting to reflect on what assumptions we bring to these problems, especially around sort of, as you talked about, the developer part of the equation, where you’re starting to think about and understanding well, if a developer starts to tackle a problem, trying to build a new service on the social web, what assumptions can they make today that they couldn’t make before? And previously maybe you made an assumption that people would have an email address before; okay, great, then you could send them a password or a token, and they can prove received that token — that’s a way of confirming… that’s a durable identifier, but you can’t really attach services to it. You can’t actually look up that email address and ask it, y’know, ‘well where is this person’s photo store’, y’know, ‘where is this…’ — there’s no directory for that. Using URLs as identifiers sort of helps to at least make that situation a little bit better. So I’m actually interested in hearing from Will, ‘cause Will and I work on the DiSo project together. How do you think the ability to use unique URLs to identify people — which then have services offered at the end of them — changes what you can sort of take for granted? What are the building blocks now, that when you approach building a new application, you’re like oh, well they probably have identity, they probably have, y’know, some service that they’re gonna use, we’re just gonna throw it all together and make it happen?

WN: Yeah, right. I mean, we’ve been trying to address those exact problems of ‘how can we build that infrastructure so that you can take advantage or take it for granted?’. And like you said, y’know, you can’t attach these services to email addresses. Recently, I co-authored a spec — and we’re now using this with a lot of our stuff — called ‘EAUT’, which is e.a.u.t. It’s Email Address to URL Transformation. It basically allows… it’s a really standard way of taking that email address and talking to whichever email provider it is that provides that, and say, y’know, ‘how can I turn this into a URL that I can then go and try to find these kinds of services on that URL?’. So y’know, Yahoo! can host their own thing, Google and all this, and then we also have this fallback service. But the idea is that, so, we can use an identifier that users are familiar with, and comfortable with, and use everyday, and they love — it’s their email address — but still be able to get these additional kinds of things. So, y’know, when I go in to whatever it is, I can give my email address, it can be converted into an OpenID; this application can then go and look at my OpenID and say ‘hey, y’know, we need to publish a bookmark, where should I do that?’ And, y’know, I could be doing that on Ma.gnolia proper, or if, y’know, I have my own Ma.gnolia instance running, with the new open source version, or if I’ve, y’know, maybe… maybe Vidoop, y’know, we set up our own, for our employees, I can say y’know what, that’s where I do mine. And so all I have to do is present my identifier, and, y’know, magic happens, basically, and this consumer can push that bookmark out to wherever it is that that I store them.

LH: Right. I think it’s important, it’s like that’s another precondition we haven’t explicitly mentioned. It’s like, we have the… we know identity exists, and we also, with the unsung but super-important spec XRDS Simple is really key to making that happen, because at least for programmers, it’s like, they can go to the end of your OpenID, and they can say ‘well, this is someone’s identity URL’, and they can get a whole bunch of links off of that. But they’re really meaningless until we have a way of saying ‘what do each of these links do?’, and so I think, I just… point that out as a key piece to that, which isn’t just having the thing, but also having the mechanism to say ‘what can we do with these different things?’.

CM: To put that another way, it’s kind of like when you go to someone’s blog and they have a sidebar that lists all their other profiles. What we’re trying to do, I think — and this is sort of what this discovery protocol is all about — is taking that list of URLs and making them make sense to computers, essentially. So that when it sees a Twitter icon, y’know, more or less, it’s like ‘oh, that’s a status update service’, or ‘that’s a microblogging service, therefore I can post messages to it, if I’m authorized to do so’. Very similarly, if you see a little Flickr icon, or if you see a YouTube icon, those are different services that someone might use that actually have APIs that you can talk to. So if you can advertise those URLs through this discovery/specifications at the end of an OpenID identifier, that’s where the magic starts to happen. So, what I was actually interested in hearing you two guys talk about a little bit — and recently, y’know, you’re wearing the WordCamp shirt, and you were at WordCamp, will, and you talked about OAuth for WordPress. and this is, I think, very interesting, because we’re at a point where WordPress currently does not support OAuth. Most of its transactions are done with the standard username and password, which means that if you wanna, let’s say, blog from your iGoogle home page, you’re gonna have to enter in your WordPress username and password into iGoogle. Well that’s great, except when you start doing this across the web and so on, and that’s the password anti-pattern. now on the other side, we have Ma.gnolia, which already supports OAuth in the platform. And I’m interested sort of in hearing you guys talk a little bit about the pros and cons and the challenges of retrofitting OAuth and authorization-based permissioning into a platform like WordPress, whereas Ma.gnolia already has that — Ma.gnolia open is gonna come out supporting that from the get-go — what does that mean for people building on these different platforms? How does that actually improve the situation?

LH: I think it improves the situation that enables a lot more seamless experience for the end user that… I think combining OpenID with discovery with something like OAuth is, y’know… this is a whole lot of hot air, so I’m ashamed of the words that are about to leave my mouth, but the browser of the future…

CM: Uh-oh.

LH: There, I did it.

?: He did it! He did it!

LH: … will be like…

SK: Tshirt’s already been ordered.

LH: The browser of the future…

CM: dot com!

SK: That’s right. Do we have that yet?

LH: … when I go to save a bookmark, instead of it saving in by browser — it will have known my OpenID already, because when I launch my browser and it’s first setup, it says ‘what is your OpenID?’ — it will have verified that, it will have discovered my online bookmarking service, and it will know where that lives, and as part of that process it will have authorized access to my bookmarking service account though OAuth, and I will have said ‘yes, this browser allowed to post things to my bookmarking service’. And so ‘save bookmark’, it will be seamlessly integrated to me. And that really is the end-user benefit for all this, despite all the horrible, geeky, completely incomprehensible nerdiness and the ongoing usability issues with OpenID, which Ma.gnolia open has cracked open again.

CM: Yes.

LH: If you wanna participate in a great little thread about OpenID usability, go to the magnolia-2-discuss group on Google.

SK: Yeah, y’know, if there’s one thing I’ve learned over the last three years of the OpenID stuff, is it doesn’t matter how open it is, if it’s not usable, it’s broken. And so that means, y’know, I think usability has to come first, and I think we have to break some things around the openness of it to get it right first. And we’re seeing, y’know, Facebook — we were talking about this today. Y’know, what Facebook is doing sort of, y’know, in the eyes of a lot of folks who are very open-centric — which doesn’t really make sense, but anyway — they see that as awful, because they…

CM: Whoa, wait; be more specific about what Facebook’s doing.

SK: Well, they’re embedding an <iframe>, effectively, on other sites — and this is effectively what Google is doing as well.

CM: That’s right, that’s right.

SK: And, y’know, that’s all well and good for the sites themselves — they don’t actually get the access to the user information — but, they can get more people and pageviews, which could be really important to them. but from an open perspective, it’s not that open, to be able to do that. And… what’s the other thing I was gonna say? God, this coffee really does — oh my god! sorry.

LH: These guys bought you Blue Bbottle before?

WN? Sponsored by Blue Bottle…

LH: I am writing without Blue Bottle here.

CM: It’s a plug for the [something]. Oh my god.

SK: It’s good stuff.

LH: But, so I mean… so I think we were getting… we were also talking about the whole convergence to this stuff, and what’s, like, what’s… the work you’re doing with WordPress, and like, where’s… do you see that headed in a similar direction? Is WordPress thinking about this?

WN: In a similar direction as..?

LH: As sort of, like, end-to-end integration…

CM: Well the interesting thing is that Weave — which is a Mozilla Labs project — is kind of in that direction, where it actually does kind of what you described, Larry, in allowing you to kind of sign in with some accounts, through OAuth actually authorize the browser to both publish your bookmarks and download ones that you’ve already saved — very much like a MobileMe for the rest of us, in a sense, for those of us who are not gonna pay Apple or whatever to do so. And all of… the entire sort of service stack of MobileMe could be more or less built on open technologies. It’s interesting, though, to think about what it would mean for someone like WordPress, or even someone like Drupals of the world and so on to really embrace some of these technologies, and to look at the opportunity that the browser, y’know, deep browser integration and web service access and offline storage, to some degree, would offer. And so I guess the question for Will is sort of around, y’know, what would OAuth mean to the WordPress platform? How would that accelerate the development of things, how might that make WordPress a different type of integral platform for publishing all sorts of different services on the web, perhaps?

WN: Well I guess, I mean… the most immediate use case, I think, that we’re gonna see with getting OAuth into WordPress is just allowing whatever service it is to publish to WordPress without needing the user’s credentials. So, this could be something like the WordPress iPhone app, this could be MarsEdit, it could be… Flickr already has a way where you can push your photos directly from Flickr into your WordPress blog…

CM: Well Ma.gnolia 2 actually offers publishing your bookmarks to your blog.

WN: Oh, does it?

CM: But right now, I believe, you have to take… it’s the password anti-pattern.

LH: Yeah. It’s the password… nobody… yeah. The major blogging platforms except for Blogger — which we don’t support because they don’t use the MetaWeblog API — use OAuth authentication. And I mean, that’s great for Google [something].

CM: And Movable Type is going to be supporting OAuth in the next release, I believe.

SK: It’s already out.

LH: It’s already out.

SK: The libraries are there, they’re in the core release… four, two, whatever it is.

CM: But still, we’re still at a point where we need that deeper integration, but…

SK: Well, yeah, and just kinda playing off that a little more is that a lot of people that I’ve seen are using WordPress as kinda more of a persistent storage of their social objects. Y’know, with Ma.gnolia, y’know, you have… there’s things going on with Ma.gnolia, but, y’know, Ma.gnolia might go away tomorrow, or I might wanna move to some other platform, so I wanna have a copy within my own control. So, y’know, I do a nightly pull or push or whatever to my blog. People do that with their Twitters — er, their tweets, y’know, they’ll do a day’s worth of tweets as a blog post.

?: My tweets are very important.

SK: Well yeah, absolutely, that’s how people feel. And whatever that…

LH: Live coverage of the Olympics…

?: [something] my addiction.

SK: So yeah, I mean… just simply using WordPress as kind of a persistent storage of these objects that are within the individual user’s control. And in order to make all that stuff happen in a secure way, yeah, absolutely, you’re gonna need a secure mechanism for pushing that in, and that’s gonna be OAuth at some point, once we get that built.

CM: And I think… well I dunno, we probably should wrap up pretty soon…

LH: I think we’re heading towards that.

CM: Yeah, well… so I guess actually we could close on some final thoughts, since this has sorta been a whirlwind discussion… and there’s much more, obviously, we could talk about. What it sounds like you’re talking about — I really like the way you framed it in terms of kind of your store for social objects, your generic store, is that increasingly we’re gonna have specific tools that do a good job at storing different types of social objects and providing metadata around those objects. So we’ll have Ma.gnolia, the Ma.gnolia bookmarks is like the WordPress of bookmarks in a sense, so you might use your self-hosted WordPress — I’m sorry, Ma.gnolia install — to host bookmarking-type things, which have certain screenshots of the webpages, maybe some tags, so on and so forth…

LH: And have access to all that’s going on out there.

CM: Right. And maybe you’ll also be able to push those bookmarks out, and also pull things in via that type of channel, because again, that channel is designed specifically for those types of objects. Then you have your Flickr, which might be a better photosharing application, or maybe you wanna use Facebook to view photos. I mean, who knows. Whatever the case is, moving these objects around into different web applications seems to be where this is gonna go, and being able to push the data around fairly [in a] fairly straightforward way using OAuth to control sort of who has access to read/write, that’s important, and then coming up with the standard protocols so that each endpoint kind of understands what kind of data is being pushed around is also a matter of import, I think, as well. So it’s really interesting to think about how we can actually move to real cloud computing using these types of protocols. So that was a longer closing from me, so what do you think?

SK: Yeah, I think… I think as we move closer and closer to having… putting users in control of their data — and I actually really like that term ‘social objects’ — because to me social networking actually isn’t something you do on the internet, it’s just a feature? And especially when you apply it to things like bookmarking services or photosharing sites, I wanna be able to bring my social network along with me. It should just be a foregone conclusion. And so to me, the work that we’ve all been doing has been headed in the direction. And, y’know… good stuff.

WN: Yeah, me too. Plus one. Plus one!

SK: Plus one for me.

LH: I think for me, it’s like, what we’re gonna be seeing next, since — is that mandatory, do you have to close with ‘what’s next’?

CM: No.

SK: Absolutely!

LH: But I’m gonna do it. What we’re gonna be seeing next is — and Ma.gnolia isn’t the first — but we’re gonnna be seeing these kind[s] of services becoming more decentralized, which means… which means creating another problem. But the ‘more decentralized’ means more reliability, more control, more adaptability to individuals’ needs. But that removes a lot of social functionality, removes a lot of community, removes a lot of interaction. So we’re gonna be seeing that problem solved. We already know how to decentralize; we do it with blogs. It’s there. But we’re gonna be seeing the federation problem being solved over the next few months. And we’re gonna see how we can bring those together in more of a ‘small pieces’ type solution [something] social network.

SK: We’re gonna solve the problem in six months?

LH: Yeah. Yeah.

SK: All right, let’s go.

CM: We’ll be there.

LH: High five.

?: Yeah!

CM: So just one more plug for… it’s Ma.gnolia.org is where you’re gonna find this stuff, and it’s ma-dot-gnolia-dot-org, that’s where you can find out more about the announcement, the m2 — as it’s being called — charter, and…

LH: And the Google group.

CM: Yeah.

LH: Come and join and contribute to the discussion.

CM: There you go.

written 14 January, 02009 Comments

Transcription of ‘Phish My Phail Whale’

Transcription of Citizen Garden episode 10, ‘Phish My Phail Whale’.

This took about four and a half hours to transcribe, and another to edit for publication — not a very good rate, but this is my first attempt at something like this. I think podcast transcription is important, if not necessarily exciting — even ignoring the accessibility issues of podcasting, transcriptions allow search indexing and make it more convenient to refer to topics that are discussed.

I’m willing to do more of this sort of thing for interesting material, though I’m not sure I’d want to go much longer than the half hour presented here.

PS: I’m not sure what would be the ideal markup for transcriptions, so I’ve made a guess based on an Adium message style I wrote that’s in turn based on an experimental conversation microformat which has since disappeared from the microformats wiki. Suggestions for improvement are welcome!

Participants

Transcript

LH: Hello, and welcome to episode ten of the Citizen Garden Podcast. I’m Larry Halff…

CM: I’m Chris Messina.

LH: And today we are joined by…

AP: Alex Payne, I’m API lead over at Twitter.

LH: So it’s been an exciting week over here…

CM: Yeah, it’s actually, it’s been a while since we did our last podcast.

LH: It’s true. But this is episode ten, which means we actually did nine of them last year.

CM: That’s true, so that’s not too bad; that’s almost once a month.

LH: Almost.

CM: Almost, y’know. That’s like a baker’s dozen.

LH: I don’t think that’s how they were distributed, though.

CM: No. Anyways. So, it’s January seventh, happens to be my birthday…

AP: Happy birthday!

LH: Happy birthday!

CM: Thank you. We’re talking today about Twitter, but in the context of perhaps a larger story around security, phishing, authorization, identity, blah blah blah, all that good stuff. Maybe for some background, Alex, you wanna tell your story or your impression of what actually happened in the two incidents that seemed to cause so much controversy over the last week.

AP: Sure. So, sometime over this past kinda New Year’s holiday weekend, we started noticing several phishing attacks going around. The first one was pretty benign, and then the subsequent ones seemed to grab a user’s account and sent around some direct messages propagating links to not just this Twitter-oriented phishing site, but also to phishing sites for Facebook and a couple of other social networks.

CM: This is, like, the access-logins.com website.

AP: Yeah, which could not be a more blatant… I mean, we were joking about it around the office. It might as well have been phishing-site.com, but…

LH: I got that one link!

AP: Yeah, I’m registering that domain when I get home, actually; backup career. But, so, it turns out phishing… we think of Twitter’s user base as getting more mainstream, but the core of it is very techy; but even against our relatively techy userbase it was still pretty successful. So, our administrators and support folks spent a bunch of time clearing out affected accounts and resetting passwords for people, trying to scrub out the phishing URLs, and we mostly put a stop to that. But just before that crisis ended, someone decided to use a dictionary attack against one of our support staff, and she happened to have a common dictionary word — ‘happiness’, since it’s been reported in the news. It’s…

CM: It makes me happy that that’s her password.

AP: Yeah, I mean, I’d love to log in with that password every day, but… so, a dictionary is just… you try every word in a dictionary against a username. It’s been an attack people have been using since back in the days of VAX systems and that kind of thing. And unfortunately it still works, and because we didn’t have any rate-limiting on authentication, we didn’t force people to solve a CAPTCHA or do something like that after they’ve logged in with lousy credentials too many times. So, we fixed that —  it’s reactive security in action! — and we’ve now got all of our support staff using strong passwords, and I’ve been encouraging people to use a great tool for the Mac called 1Password, which lets you generate strong passwords, store them, that kind of thing. So, we’re talking a bit more internally about building security into our day-to-day practice. It’s something that I think is really difficult for a fast-moving business to do. You wanna spend your time building exciting user-facing features, not locking stuff down, but it’s just the reality of being on the web. There’s lots of bad folks out there.

CM: Now, what kind of background would you say you have in security issues and things like that? What kind of experience are you bringing to this current situation?

AP: I’ve kind of bounced back and forth between doing web development and doing security stuff. Actually, one of my very first jobs when I was still a teenager, I was a web developer, and the company got broken into, someone decided to…

CM: … physically, or..?

AP: Electronically.

CM: Okay.

AP: So, someone decided to use our server to hang out on chat rooms in Lithuania or something, and trade warez and that sort of thing.

CM: There’s like ten people in Lithuania, so it’s…

AP: Right, and they were all on this IRC channel. So, I got a very rapid education in, y’know, all right, this is how we wipe and reinstall machines, secure it from the ground up. I got really interested in intrusion detection, all that security stuff that big in the ’90s, and just kinda continued down that road. I spent a couple of years working for a sorta information-security–oriented government intelligence contractor, and there I continued to do sort of a mix of web stuff with towards security. And as a hobby around that time, some friends of mine and I helped run the hacker game at the DEFCON conference every year, ‘capture the flag’. So, we competed in that one year, and then this group of friends took it over, and they’re still doing that. Most of them are back on the east coast. That was fun. I got to write web apps and see if they stood up against the best hackers in the world, and that sort of thing. So, security’s been both a hobby and sort of a professional thing for me, from time to time.

CM: so that brings up an interesting point… I was sort of aware of some of your background in security stuff. It kind of leads into a question about Twitter, of course. I mean, if this was your hobby, and you’re the lead of the API stuff… Twitter is more or less a porous application. I mean, it’s been reported, actually, I think in 2007, that most of your traffic comes from off-site sources — meaning that people are not coming to twitter.com, necessarily, and interacting with the service; they’re doing it from Twitterrific or from other applications, or third-party websites like [Hahlo?] and things like that. So I guess, the first part of this question is ‘how secure is Twitter’? I mean, when people are using it and so on… if you were back playing capture the flag, how long would it take you to capture the Twitter flag?

AP: Well, I think there’s sort of a couple of things. The first is that most of the information on Twitter is designed to be open. People can have protected accounts… my personal opinion is that I wish we didn’t have that feature. I feel like people get so much more out of Twitter when they have a public account. I know that there are some people who just aren’t comfortable with taking their thoughts or taking their social network out in the open. So we accommodate them, but it’s a relatively small part of our userbase. And one of the main problems we’ve had over the past couple of years has been ensuring that protected accounts never leak. We’ve definitely had points at which our API has inadvertently exposed people’s protected tweets because our code has to accommodate this complicated privacy intersection of ‘user a is looking at user b’s friends, which can include user c, who may or may not have authorized user a, but has authorized user b’. And with all the caching logic and stuff in the middle of that, there are bound to be bugs, and there have been security problems around it. That’s, I think, the biggest issue we’ve had with Twitter. I’m pretty confident, given that we’ve got two different test suites now for the API, one sort of baked into our application and one completely external, that look for some of these security issues… and we’ve hired other kind of security-minded folks — one of our hires last year was John Adams, who was a member of the l0pht hacking group back in the day…

CM: He also wrote some of the Declaration of Independence, I heard… sorry.

AP: Took me a second. So, between John and I and the other folks, we’ve tried to sort of, in our spare time, look at Twitter from a security perspective. I wish that we could say that we’d done a full security audit and that we’d brought in an outside team — that’s one of the things I’m hoping we can fit into our schedule sooner rather than later.

LH: One of the interesting things is the phishing attack that happened. It’s sort of like, outside of the realm of a lot of the standard security procedures.

AP: Oh, sure.

LH: And it’s like, very… securing your site against, like, engineering-type hacking is very different than trying to secure your against social-type hacking. And it’s really easy to hack around sites where you’ve tried the security and social hacking. So, for instance, it was one of the things — I think it was even someone [who] was at the OAuth summit, or was it the OpenID summit? — they were talking about one of the new anti-phishing things is showing… you pick an image when you log in…

CM: Site seal.

LH: … the site seal idea, which a lot of banks have now. But they found basically, you can trick people out of that almost all the time by saying ‘site seal is not available at the moment’. So people read that and say ‘oh, well I can’t get my site seal, but I need to get to my bank account, so I’m gonna sign in anyway’.

CM: You should just put, like, a broken image or something like that.

AP: And I mean, certainly the fact that most of our longer-term users know that we do occasionally enable and disable features depending on traffic and that kind of thing; they’re used to saying ‘well, I can’t get to this part of Twitter right now’, so I don’t know how effective that would be. This other social kinda web-oriented security thing that’s come up is just… we’ve had problem after problem with sharing stuff via JSON, having callbacks, and… you’re trying to support mashups… a couple of folks — older, kinda comp. sci. folks — that I follow on Twitter have joked at times that from their perspective, the whole social web is kinda one big cross-site request forgery attack. Y’know, if you’re not involved in the social web culture day-to-day, where you’re excited about this stuff — if you come at it from more of a privacy perspective, it’s like ‘this is really scary’. Y’know, the whole mashup thing is basically… it takes advantage of the fact that browsers still have a pretty primitive security model. I sorta wonder how much of the mashup culture if people went back to the drawing board with browsers; went back to kind of…

CM: Well, if they did it the right way.

AP: Yeah, basically.

CM: Yeah.

AP: Yeah, so, we’ll see. And people having to come up with really complex solutions for that.

CM: Yeah.

AP: Like Yahoo! sort of turning Yahoo! Mail into this host for JavaScript apps, mashups, and that kind of thing. But they’ve had to implement…

CM: Basically they have to rewrite JavaScript.

AP: Yeah, Ben Laurie’s Caja project, which is a whole capabilities-based secure reimplementation of JavaScript. So, y’know, that’s pretty heady lengths to go to just to be able to support mashups. But over and over again, we’ve had people point out that ‘Hey, I can get to this data via Javascript, and a malicious site can control it’ because some old browser allows them to redefine the array data structure, or take control of callbacks, or that sort of thing. So that’s another area where we’ve had to be reactive and not proactive because, y’know, the community has been out there finding all of these bugs that you wouldn’t find if you sat down for a hundred hours.

LH: There was sort of…

AP: And the other thing is…

LH: And the angry mob sort of misguidedly decided that the solution to the phishing attack was the long-awaited Twitter implementing OAuth. So the meme went around of, like, ‘Why hasn’t… I’ve been phished; why hasn’t Twitter implemented OAuth?’.

CM: Well, the password anti-pattern became kind of a household word, like, over the past two weeks.

LH: It was like… people… it was pretty wrong, because OAuth is not… would not secure against phishing.

CM: … would not have solved these problems.

AP: Right. Yeah, I mean, it’s become so much of a household concern that I was talking to a reporter the New York Times about it the other night, and she was asking for sort of a layman’s explanation of OAuth — but, thankfully, she read enough of what was going around on the web that day that her angle on the story wasn’t ‘if only Twitter had put out OAuth there would have been no phishing.’

LH: Wouldn’t that be great if OAuth had solved all phishing?

AP: Or OpenID, for that matter.

CM: It basically… it made everybody like fifteen IQ points smarter. It’s amazing! It’s amazing what the internet does.

LH: I mean yeah, so it’s like, I think that’s the problem, it’s… when people start doing armchair security…

CM: Let’s talk about that, though. that’s… I think on the one hand, there’ve been, y’know, some of us geeks that are out there, sort of pushing the meme of the password anti-pattern, because it is actually something that people should take seriously. More from the perspective that, y’know, just like with good… what I call ‘data hygiene’, you should be checking out the URLs of the sites that you’re visiting and so on to make sure that they are actually Facebook and not facebook.access-login.com, you should be considering where you put your password in on the web. And one of the problems we’ve seen is, of course, that with Twitter, on the one hand, you can make the argument that no, it’s not your bank, so you’re not gonna go broke if someone hacks your account —  but on the other hand, there’s a point to be made that Twitter stores a great deal of what I call sort of ‘data capital’ or ‘social capital’, and that if someone took over one of our accounts and we’ve been tweeting for a while, and we have some followers, that all of a sudden large number of people immediately are going to see something that we can’t take back. Y’know, it goes out over SMS, there’s no takebacks. So, this happened, of course, with Barack Obama’s account, y’know; a hundred and sixty thousand people, let’s say, receive an SMS from Barack Obama saying, y’know, go check out, y’know, this survey and get a free iPhone or whatever. And that’s a very strange thing to get from the president-elect.

AP: Sure.

CM: So, I think that the real consequence of not providing people who have, let’s say, accrued that social capital or data capital the opportunity to secure their accounts somehow means that there will be these mobs that say ‘Why didn’t you do this? This could have been prevented.’

AP: Sure.

CM: And the answer is, well, no, maybe it actually could not have been prevented, because people were tricked, and this technology doesn’t prevent people from being tricked. But by the omission of not doing it, you gave people that opening to complain and bitch and moan.

AP: Sure.

CM: And so one of the things I wanna talk to you about, though, specifically, is I know you’ve had some criticisms about OpenID and OAuth; you sound very skeptical of them. You’ve also said that the technology is not… even though Blaine and Twitter were part of the creation of OAuth — in fact Larry was the other sort of 50% of that — Twitter still does not offer that. So it would not have… you can’t say, ‘Well, they did have OAuth and the problem happened anyways, so clearly the OAuth was not the solution. Oh! the solution must have been to make people smarter.’ Instead, it fell onto OAuth. So I’m curious, what’s your thinking about that, and, y’know, what’s next in terms of this conversation?

AP: Sure. So, the road to OAuth has been kind of complicated for us. Blaine sketched out a prototype of OAuth in the Twitter codebase. It was there for a while, but like our early API, it was present but not documented. And a handful of developers, mostly via word-of-mouth from Blaine — folks like Kellen over at Flickr — started building Twitter applications that talked to this early sketch of OAuth.

CM: He’s one of the authors of flickrauth and one of the early writers of some of the OAuth [something]

AP: Sure.

LH: And the OAuth validation was wrong. It doesn’t…

CM: That’s right.

LH: It signs differently than OAuth does.

AP: Right, right. I mean…

CM: So it was there, it was wrong, it was taken away.

AP: It was a pre-spec implementation. I mean, it wasn’t even called OAuth per se, y’know; all the internal code had no reference to OAuth or the nomenclature of OAuth. And so at a certain point in the middle of last year, we said okay, there’s a real spec. I sat down to implement it. I took out Blaine’s old code, started putting in the new code, and was concerned about the quality of some of the Ruby code out there to handle OAuth, and at that point in time, working on the Twitter API was essentially what I did in my spare time after not sleeping after not working on other Twitter features…

CM: Which creates, usually, very secure code, y’know.

AP: Right, exactly. And so at the time, I was terrified to implement this kind of half-baked support for OAuth when the API wasn’t my full-time job, we hadn’t had time for a proper security review, and then talking to Eran Hammer-Lahav, who’s been a big voice on the OpenID spec. He happened to be in San Francisco, and talking to him, he said well, there’s a couple of things in the spec right now that we still wanna iron out; you shouldn’t necessarily put a full stop on implementing OAuth, but if you wanted to wait several months, you’d be implementing the sort of most secure version of the specification. I guess there was some chance that a timing attack could be accomplished…

CM: Do you know when that was, when you had that conversation?

AP: I wanna say this past summer.

CM: Okay.

AP: So with all those factors convening, we just decided to table it for a while, and at the end of last year, as Twitter had grown a bit, I was given the role of API lead and allowed to focus on that full-time, and was given one other employee to work with me on the API — who, up until recently, has been doing administrative duties on our search site. But getting the OAuth implementation done has been his project and now that most of his other work is behind him, that’s why it’s finally happening.

CM: Yeah.

AP: So that’s why it’s taken so bloody long just to get it out there — it’s just a matter of internal priorities. My opinion about the technology is kind of separate. I haven’t been dragging my feet on OAuth because I’m concerned about its threat profile or something. I agree completely that it’s a step forward from basic auth and having people submit their username and password, but mostly I’ve just wanted to make sure that it’s deployed with the same level of care that we’ve tried to deploy everything, particularly since kind of mid-2008 when we’ve really just tried to make a push to turn Twitter’s reputation as an unstable service around. And it seems like following that kind of slower development model has worked out pretty well — people don’t talk about Twitter’s instability quite as much.

CM: The fail whale just doesn’t show up quite as much.

AP: Yeah, exactly.

CM: We miss it dearly, but…

AP: So…

CM: But, y’know, I mean… not to [mislead?] or completely throw your words at you, but you did say — or tweeted, at one point — that, yes you’ll do OAuth at some point, but users and developers are gonna hate it.

AP: Yep.

CM: And that, that’s…

AP: I stand by that, actually.

CM: And I really wanna get your opinion on it, because one of the things that hopefully we can do is take constructive negative feedback and turn it into something that results in a number of bug issues that can be corrected.

AP: Right. And I mean, obviously it’s hard to fit a nuanced criticism into 140 characters.

CM: … in 140 characters, that’s right.

AP: So, when I say something like that, it sounds definitive, it’s not saying ‘they’re gonna hate it and we’re utterly unwilling to work with folks like Chris and Larry’…

CM: It just comes out that way.

AP: So…

CM: … in 140 characters!

AP: Yeah. So… some people’s rebuttals have been, y’know, similarly curt, and it’s just… it’s just a fact of the medium. And we all kind of enjoy throwing slings and arrows back and forth.

CM: Twitter is the new godwin’s law!… all right, go ahead.

AP: So… this isn’t entirely my personal opinion. The approach I try to take to developing the API is that we deliver API methods that mirror the features that we expose on the site, and we deliver API methods that are in direct response to what our developers want. We don’t… I don’t think we’ve ever said ‘here’s a method or an API feature…’

CM: … that no one wants.

AP: ‘… we’re going to deliver because we think it would be a great idea’. We always try to do things that the community’s asking for, and that’s part of what’s kept OAuth on our priority list. But the feedback that I’ve heard from developers who’ve implemented OAuth for other services has been pretty negative. People aren’t crazy about the quality of the libraries that are out there, they’re frustrated by the user experience issues — a lot of developers, particularly on mobile platforms, just kind of don’t know how they’re gonna handle OAuth in an elegant way. And, I guess, from the user’s perspective — I’m basing that a lot on the user studies we’ve seen about OpenID, which… users hating it is maybe an overstatement — but users being confused by it seems pretty fair — that seems to be like what the Yahoo! team found out.

CM: … with seven people.

AP: Yeah, with seven people.

CM: Y’know, I could survey my family and get the same response.

AP: Well, but that’s just it — chances are pretty good that, y’know, if you asked your mom to do the OpenID workflow as opposed to just putting in her username and password, it’s more confusing. That doesn’t mean, y’know…

CM: If you surveyed seven Twitter users, maybe they’ve used Facebook and they’ve authorized an application, for example.

AP: Maybe, yeah. I mean, we’ll find out. To me, it still seems like OpenID, OAuth are inevitabilities; it’s just gonna be a bumpy road.

CM: I mean, absolutely. Part of it is — and we talked about this briefly —  it’s like, y’know, Facebook Connect is great, ’cause it gives you a nice, y’know, bluish-purple button that, y’know, people can click and say ‘oh, I recognize that, Facebook Connect. Lovely.’ The problem comes when you wanna have choice, and when your friends are not necessarily on Facebook, but they’re on MySpace or they’re on Twitter, and you wanna bring your friends with you. And I think that’s one of the real hurdles that eventually, y’know, no, not everyone’s gonna be on AOL, y’know; they’re gonna wanna have their own email account some place else, and now you’re gonna have to figure out that interop. And you can imagine that at some point, somebody said ‘People will never understand email addresses. What is this ‘factoryjoe at aol’ nonsense? It’s just, y’know, it should just be like the single name.’. But then, of course, we figured out a way to move forward [from] that, and there was enough value there where people could… essentially internalize the notion that someone might actually be on a different server, and there was some way of referencing them.

AP: Right.

CM: Now, I mean, I’m curious then — what, first, what are your recommendations for people who are using Twitter? — y’know, what should Twitter users do in terms of securing their account — but also, what are some implementations of this type of… these alternative security approaches that you’ve liked? Because sure, you can say that the OAuth user experience sucks today and the OpenID user experience sucks today and that sort of sets the baseline; we can only get better from here… what are good examples that, y’know, are inspiring you?

AP: Right.

CM: … in terms of solving these problems.

AP: I don’t think that there are particularly inspiring patterns for security on the web. I mean, y’know… SSL — we were talking about this before we started recording —  SSL gets the job done, but when you start using SSL with certificates and all that kind of thing, you really need a kind of a organizational administrator to handle most of that stuff. So maybe a Fortune 500 can roll out SSL with certificates to all of their desktops connecting to their intranet or something like that, but we can’t realistically expect most, y’know, home users or users of a mobile device to make use of SSL. So I’m not sure today there’s a pattern that’s better than what OAuth suggests. There are a handful of Twitter apps out there that talk basic auth over HTTPS.

CM: Yeah.

AP: And that seems to be working pretty well for iPhone apps, for desktop apps…

CM: That doesn’t solve the problem of giving your credentials to somebody else; just that it’s less likely that they’ll be intercepted.

AP: Yeah. I mean if you trust the application, and…

CM: Right.

AP: … and you know that the credentials are only ever being stored on your own computer…

CM: Like, if you trust the application, like Twiply or something.

LH: Or Twitterrific.

AP: Twiply was web-based, right?

CM: Yeah, sort of joking… the one that was sold, and like ten hours after it collected eight hundred user accounts.

AP: Right. But in the case of desktop apps and iPhone apps, there’s some that are open source. There’s spaz, which is an open source Air app that runs on everything, Twitterfon is completely open source… so really paranoid users can audit the code for that sort of thing. That’s not a great solution for, y’know, mom and dad.

CM: There’s, y’know, there’s some other benefits, though, I think, of moving to this alternative model that arrive over time. One is rate limiting. Y’know, if you have these applications making use of the Twitter API and you have really no way of identifying them besides IP address… it would be nice if you could just shut off a class of, y’know, malfunctioning applications. Like, specific desktop applications, let’s say, that just do terrible things. You also get the benefit of having kind of a paper trail, of saying ‘these applications changed your account in some way’.

AP: Sure.

CM: And, y’know, a similar idea that Ian McKellar came up with just the other day was this notion of pushing changesets to your account, and then you could approve or deny them. It might be a little heavyweight and a little awkward, but, y’know, similar sort of idea, where instead of using consumer keys, it’s just like ‘I have this changeset, I wanna deliver this parcel to your account’, so at some point later on you come along and approve that, and it goes through.

AP: Right.

CM: So there are different methods there to solving this problem, but clearly, I think, there’s a balance between the complexity on the user side and what hurdles they have to jump through to have sort of a more secure experience where they’re not handing out their credentials, as I’ve said before, like a [something]. As well, you have to think about the developers, and make sure they’re not tasked with something that’s so arduous that it’s impossible for them to implement.

AP: Right.

CM: And it’s interesting to, y’know, ’cause you are gonna be — and you can speak to this in a second — you are gonna be rolling out OAuth as sort of a beta release soon. There’s a service called brightkite which lets you set your location or whatever, and they’ve done an interesting thing where, for developers testing on their sort of staging site or whatever — y’know, just testing applications — they allow them to use basic auth, just a way of trying out their stuff. But if you wanna actually interact with user data, then you have to move over to OAuth. And so there’s a nice sort of balance there where you can’t do any real damage with usernames and passwords…

AP: Right.

CM: … when you’re just trying things out to make sure your app works. But then when you wanna get into the real deal, then you actually have to go through the proper sort of, y’know, dances to make that happen. So…

AP: Right.

CM: I’m curious, in terms of, also… maybe an extracted question is: where are you with OAuth? When do you expect it to sort of land? What’s your process look like? And what kind of things are you working on right now?

AP: Sure. so… where we’re at with OAuth is that we’re very close to a private beta — by private, that pretty much means we’re gonna post a message to the Twitter development-talk group, and anyone that says they’re interested and looks capable of giving us decent feedback will get in. We’re not gonna super picky about it — fifty, a hundred people, that’s fine. So, we’ll go through the private beta for maybe two, three months, unless there are some glaring faults; then move into a public beta for another couple of months; then when OAuth [support?] is final, we’re gonna have six months during which we encourage developers to migrate their apps to OAuth, and at the end of that basic auth will be deprecated. And depending on how that release goes, with rolling out the next version of our API in general, we may put new API methods only behind OAuth, or bump up the rate limit for applications that OAuth, or just try to find ways to really incentivize developers to move over.

LH: That’s what we did with ma.gnolia, is like, you get… in order to get any of the stuff that’s new and version 2 of the API you have to use OAuth.

AP: Right.

LH: But you can still use all the old stuff with version 1 with the two older auth methods.

CM: And that’s sort of an interesting balance there, where, y’know, if you wanna do basic auth, you have the most basic functionality whatsoever. Y’know, you can post a message a day, or something… or thirty messages a day, or something reasonable, where if you’re lazy or an inexperienced developer, you can still create an app for your friends, but if you wanna put something out there that people are gonna download and use and install for a while, or put it on their mobile device — which could get stolen or whatever — then you actually have to go with something that will support that type of functionality, namely, using OAuth.

AP: Right. And I think one of the ways to… the nice thing about basic auth is that people can read the Twitter documentation…

CM: It’s basic, as you’ve said.

AP: Yeah. They can use curl and just poke at API methods, see what they get back. In the absence of that, we’re gonna need to provide a nice sort of interactive sandbox for the API.

CM: Yeah.

AP: And that’s something we wanna do in the next version, so you can test out a URL, see sample data, that kinda thing. But… I’m looking forward to what we’ll be able to do when we have OAuth — there’s a couple of things we’ve been wanting to be able to do for a while. We’ll be able to disable malicious applications from our end, so if we find a Twiply or something like that, we can just say ‘all right, for all the users that have this installed, it’s gone.’ Another nice thing is that we can build an application directory — right now, that’s sort of informally maintained on the Twitter fan wiki, and on a couple of other sites. but since every developer has to register every application, we can ask them for a little bit more information about their app, and we’ll have enough to build a pretty gallery where users can say, y’know, ‘show me all the mac apps’, ‘show me all the blackberry apps’, that kinda stuff.

CM: That’s pretty exciting. Now, it also suggests — and, y’know, this is sort of my interest, since I was recently elected to the board, but uh… — that it opens the door, at least, for Twitter to consider finally maybe doing OpenID at some point as well. Is that at all in the frameworks, or should I just wait another year, and once OAuth’s done then we can have this conversation again?

AP: You should probably not hold your breath. I mean, this is one of those prioritization things. I mean, right now, you can see all the issues that the API team at Twitter tracks — we keep all that open — and you can see that the OAuth issue has the most stars on it, has the most votes on it. But there isn’t an issue for ‘Twitter really needs OpenID’, because…

CM: Let me ask you something…

AP: … nobody’s asked, other than yourself.

CM: Other than me!, yeah. Well, y’know, I tend to be the outlier. How about delegation? We can start there.

AP: That, I think would be worth maybe…

CM: ’cause then I could use my Twitter account as my OpenID…

AP: Yeah.

CM: … without you guys having to do any work.

AP: Yeah. That I think would be worth doing; I think it would probably end up being Britt and Rael and the UX team that ends up implementing it; it’s not really API-related…

CM: Fair enough.

AP: We’re doing enough specialization at this point that, y’know, I can kinda say it’s their problem. I honestly think that’d be nice. At the very least, it would be nice if people can log in with their OpenID and make use of it. Twitter as an OpenID provider — I think we’d want to have a really great handle on the kind of phishing problems out there before we did anything like that — but you should be able to log in with your OpenID. That seems perfectly reasonable.

CM: Awesome. Well, that was great, then.

LH: Yeah. I think we’re gonna wrap this up?

CM: Yep.

LH: Thanks for joining us.

AP: Sure, thanks for the opportunity.

CM: Yeah, appreciate it.

LH: And we’ll talk at you next time.

written 10 January, 02009 Comments