The joy of Perl

Originaly from Salon | 21st

BY ANDREW LEONARD | Larry Wall smiles when he recalls the message that Yahoo co-founder David Filo sent him several years ago, shortly before Yahoo was to go public. Yahoo, wrote Filo, could never have been started without Perl, the all-purpose programming language Wall invented. So would Larry like to buy some cheap, pre-IPO stock?

Back in early 1996, at the absolute height of Silicon Valley Internet IPO madness, such an offer was akin to asking if you would accept a dump truck delivery of solid gold ingots on your front lawn. But for Wall, money has never been a primary motivation. Though widely acclaimed as the author of one of the most valuable tools for hackers anywhere, Wall lives modestly in suburban Mountain View, Calif., tooling around town in a well-worn 1977 Honda Accord. Perl itself was never about money -- Wall created the language to solve a programming problem he faced during his day job, and from the get-go he made sure that the source code to Perl would be freely available. People are always allowed to tinker with Perl -- regardless of whether they use it to construct a multibillion-dollar Internet directory company or just to get a survey form working on their own home page.

Still, Wall may be frugal, but he's not stupid. He accepted the offer and bought some Yahoo stock for his 14-year-old daughter -- enough to pay for her college education. A better example of the Internet's old "gift economy" ethic could hardly be imagined -- give unto the Net, and you shall receive.

Larry Wall likes to call Perl a "humble" language. In his soft-spoken voice, he describes Perl as if it were a meek, obeisant servant, existing only to "let you bend it to your uses." The legions of Perl hackers who swarm the Web are less modest: Perl, they declare, is the indispensable duct tape, or glue, that holds the entire Web together -- not just Yahoo, but Amazon and a million other sites. Without Perl and Larry Wall, Perl's advocates argue, the Net would be but a pale shadow of its current self.

Wall has played an important role in spurring forward not only the Web's evolution but also the burgeoning free software/open source movement responsible for so much of the Internet's structure and plumbing. But even though his peers hail him as one of the "paramount chiefs and wise elders" of free-software culture, Wall's version of leadership is utterly self-effacing -- a character trait that sets him apart from some of the other leaders of the movement.

The son and grandson of preachers, Wall is himself a religious man. His mission, says his friend and close Perl collaborator Tom Christiansen, is to act on his belief "in people working together. He projects his inner vision of selfless work for mutual benefit onto Perl." But he won't allow himself to be drawn into the petty "religious" wars that plague the world of programming -- the endless disputes over such issues as whether one programming language, or operating system, is inherently better than another. Such flame wars are a favorite hobby for hackers who are fond of declaiming in terms that allow no wiggle room -- ambiguity being foreign to the fundamental either/or, yes/no, zero/one digitalness of the computing universe.

But Wall and Perl are all about wiggle room, about messy imperfection and fuzzy creativity. After all, duct tape is valuable not because it offers a perfect solution to your plumbing problems, but because it gets the job done. Perl, to some eyes, may not seem elegant. But that's not Wall's concern. His humble goal is to be useful, to help people do what they need to do -- to facilitate the interconnection of programming languages, hardware platforms, assorted software universes and people working together into one cosmic entity.

Which, if you think about it, is what the Web is all about, too. It's no accident that Perl, which Wall first invented more than a decade ago, didn't really start to explode until the Web took off in 1994. The Web is a hacked-together, messy, ad-hoc creation that requires fast thinking and faster reaction times. Perl is a Web hacker's best friend.

"I try to get the right amount of levity and gravity simultaneously," says Wall, sitting in the moderately cluttered living room of his Mountain View home. He's talking specifically about Perl, but his attitude is built into the infrastructure of his daily life. Wall has Caller ID hooked up to his computer, which he has instructed to emit unique sounds for each incoming call. What good is Caller ID, asks Wall, if you have to walk all the way over to the telephone to see who is calling? Likewise, pressing his doorbell sets off a sequence of dialogue from a Wallace and Gromit claymation cartoon, and when his dryer -- hidden away in the garage -- finishes its cycle, a whistling sound shoots through the entire house.

Wall's professional background is as a system administrator, rather than as a software engineer. "Sysadmins" tend to take a utilitarian attitude toward programming and technology -- they focus on keeping balky networks running, solving pressing problems now, hacking on the fly. Wall's whole life as a programmer has been devoted to solving those kinds of problems -- Perl is just the most recent tool in his personally crafted arsenal. Years before he dreamed up Perl, he had already achieved hacker renown by writing "rn" -- a program for reading Usenet newsgroups.

Rn was an early prototype for what is now called the free software or open source model for software development -- in which far-flung programmers collaborate across the Net on improving products whose code remains accessible to all. Wall wrote the rn program, released the source code to the Internet and then started work on an upgraded version that incorporated suggestions and bug fixes from fellow hackers all over the then-fledgling universe of cyberspace.

But back in the mid-'80s, it wasn't easy to distribute upgrades across the Net. Often, people would be connecting via slow 300- or 1,200-baud modems, and they simply could not fling megabytes of source code back and forth in the carefree manner that is now commonplace.

So Wall wrote a tiny program called "patch." Patch took a compact new code upgrade and applied it to old source code. Patch could bring that old code up to speed, and was even smart enough to take into account changes that had been hacked into the old code.

As hackers go, Wall is a reasonably circumspect man, but that doesn't mean he is unfailingly modest. "Patch," says Wall, "changed the culture of computing."

"I've believed for years that patch is his single most important contribution to the open-source culture, though it's never attracted the kind of notice that his bigger, sexier projects, like Perl or rn, have," says Eric Raymond, one of the most vocal leaders of the open source movement.

"Patch may just be the most successful hack of all time," says Raymond. "Larry effectively created, or at least critically enabled, the modern style of highly distributed development exemplified by Linux."

Perl came after patch. On the surface, it's a completely different beast, a highly complex "scripting" language aimed at programmers who need to write short bursts of code that automate tasks, connect incompatible programs and systems and otherwise solve persnickety problems. Perl's inception dates to 1986, when Wall worked as a system administrator for a subsidiary of Burroughs. At the time, he was engaged in a "secret project for the NSA" that involved synchronizing information exchange between computers in Santa Monica, Calif., and Paoli, Pa. But the NSA didn't just want the information synchronized -- it also wanted reports generated about each exchange, and none of Wall's existing tools seemed to fit the task. Getting everything together required a jury-rigged solution, a hack, that eventually became Perl -- the Practical Extraction and Report Language.

"I realized at that point that there was a huge ecological niche between the C language and Unix shells," says Wall. "C was good for manipulating complex things -- you can call it 'manipulexity.' And the shells were good at whipping up things -- what I call 'whipupitude.' But there was this big blank area where neither C nor shell were good, and that's where I aimed Perl."

From that point on, Wall focused his spare time and energy on exploiting the need for a language that could connect the dots across all the big blank areas in the computing universe.

"People are always looking for the interstices," says Wall. "They are always looking for the new ecological niches. And the speed with which you can move into those ecological niches is really important, because the first person into a niche is often the winner."

Perl was a winner. Wall ensured that Perl's particular strong points -- text processing, flexibility and a vast tool kit of built-in features for quick-and-dirty problem-solving -- made knowledge of Perl essential to programmers in a hurry: people like David Filo and Jerry Yang, the two Stanford University graduate students who created Yahoo. They found Perl indispensable. They used it to generate their Web pages, write the code for their data-collecting Web robots and maintain their database of Web addresses.

"We relied on it a lot," says Filo. "Because of the quick development time, you could do things really fast."

Yahoo's experience was far from unique, says Tim O'Reilly, founder and CEO of the computer book publishing company O'Reilly & Associates. Not only does O'Reilly publish the bestselling Perl computer books, but it also hired Wall three years ago to be a full-time evangelist for Perl.

"Perl is great for winging it," says O'Reilly. "Perl is much more accessible than a traditional programming language, and I think that that was a key part of what enabled people to start building and envisioning the Web. It let the amateurs in -- you didn't have to be a professional programmer ... Even though Perl can be difficult and has a reputation for sort of being obscure, it also is pretty accessible for the kinds of things that people want to do. You can throw things together quickly and get work done."

How exactly does Perl do it? How is it different, really, from the host of other computer languages vying for supremacy under the digital sun -- Java or Python or Tcl?

Perl, says its fans, is a "Swiss Army chain saw" -- a tool that can do anything with brute efficiency. One doesn't have to be the kind of programming genius who can think in Java or C++ to be able to use Perl, although its own flexibility can make it confusing -- especially to hard-core programmers who are accustomed to having just one right way to do something. But Perl's greatest advantage lies in the way it functions as a bridge between other computer languages.

"Perl has always been, and continues to be, the archetypal glue language," says Chip Salzenberg, one of Perl's leading code contributors. "Its evolution is, by design, driven by the needs of real programmers as they face the day-to-day challenges of making complex and uncooperative systems work together."

Larry Wall believes that this evolutionary process mirrors how the real world works.

"Perl does a lot of hand-holding," says Wall, "and gives very good feedback on what it thinks is wrong with your program, so there is a very rapid turnaround if you are trying to develop something quickly. You try something and it breaks and you fix it, you just grow it, evolve it. This is how I program, this is how I write. This is how a lot of people write, this is how they think."

Perl's reflection of real life -- its humanism -- is, says Wall, built into the deep philosophical structure of the language. Perl is "the first postmodern computer language," says Wall -- "an artificial natural language" that imitates how programmers think.

The deep structure of all languages has always been a central concern for Wall. Before he committed to a lifetime of systems administration and associated hacking, he and his wife were graduate students in the linguistics department at UC-Berkeley. Their plan, says Wall, was to become field missionaries dedicated to assisting Bible translation. They would go live with a tribe that had no written language, learn it from scratch, write it down and then help translate the Bible into that language.

He and his wife abandoned that mission, but he replaced it with another -- assisting other people in the ever-relevant goal of connecting with one another to do useful things. When it came to creating a programming language, instead of building something totally new from the ground up, Wall -- inspired by his linguistics training -- chose to build something that replicated how real humans thought and acted.

First and foremost, that meant providing a number of different approaches to solving every possible problem -- which led to the Perl battle cry, "There is more than one way to do it."

"If you think of a human language as an artistic medium," says Wall, "it has to give you room for creativity. If you want to be able to optimize for different things -- if you want to be able to write recipes and poetry and newspaper columns or magazine articles in the same language -- then there has to be flexibility. This is the antithesis of what people are taught in computer science. People are taught that if there is any redundancy at all that that is evil, wicked. Coming from a natural language perspective, I just don't buy it."

Perl evolved the way a real language does, assimilating chunks of other languages over time -- a little Unix, a little C++, a little BASIC, a little this and a little that. And if anyone else came up with something new and useful, Perl would incorporate it. When Wall invokes the concept of "postmodernism" he is referring to Perl's eclectic, inclusive approach -- its ability to find a little bit of truth in every nook and cranny -- without at the same time pinning itself to any one grand Truth.

"When it comes down to it, most computer languages try to make the programmer think like a computer," says Wall. "I tried to make the computer think a little more like a programmer, with some degree of success. The question is, who's to be master, that's all."

Despite Perl's popularity, the language is not without its detractors. Marc Ewing, chief technical officer at Red Hat, the leading distributor of commercial versions of Linux, prefers working with Python; and Infoseek, the Internet search engine company, uses Python for its internal development work. Perhaps the most pointed criticism comes from Eric Raymond, who has long touted Perl as one of the great success stories of the open source movement.

"The design of the language is showing its age and significant signs of bloat," says Raymond. "Perl was never a pretty or elegant language; its lure lay in its brute usefulness. Over time, the problems consequent from Larry's early choices seem to me to be accumulating and reinforcing one another just a little faster than the benefits pile up."

"These problems have created a significant brain drain away from Perl to other scripting languages," says Raymond. "I'm part of this trend personally; I have essentially abandoned Perl for Python, a younger language with a similar application range but a much cleaner design ... I expect that Perl's relative importance will probably decline in the future, even while its developer base continues to increase in absolute size, because its competitors will grow faster than Perl does."

Perl is anything but clean -- indeed Perl hackers use the word "messy" almost as a point of praise. But for Perl's detractors, and especially for Python devotees, messiness is not next to programming godliness.

"One complaint about Perl that I hear frequently from Perl users who also use Python -- and who often but not always prefer Python -- is that even your own Perl code is hard to read back when you've been away from it for a while -- let alone someone else's Perl code," says Guido Van Rossum, Python's principal author. "This is much less a problem with old or unfamiliar Python code. While Perl proponents make big efforts to encourage Perl users to write clean Perl, and newer Perl versions contain features to help them, Perl's insistence on the value of messiness means that it will always be a problem with Perl."

On one level, the Perl/Python debate is just another example of the endless jockeying for supremacy that characterizes hacker discourse. But the core tension is real -- a tension between those who seek purity and those who are content with utilitarian efficacy.

"Language purists have always said that Perl was ugly," says O'Reilly, "that Perl was a 'kitchen sink' language, that Perl would collapse under its own weight. Yet Perl goes on solving more problems than all the other scripting languages put together."

"We sell the bestselling books on both Perl and Python," adds O'Reilly, "and what we see is a continued healthy growth in the use of Perl, while Python use is flat. Perl is orders of magnitude more widely used than Python, and we see no signs of that changing, at least based on book sales."

"Perl is not suffering a brain drain of any magnitude," adds Salzenberg. "At most you could say that other languages are growing at the same time that Perl continues to grow. And that's fine for Perl. A glue language is most useful when there are lots of things that need gluing ... Perl will continue to become whatever is needed to connect everything to everything else."

Wall lets others man the defenses against Perl's assailants. When I asked him for a response to a point-by-point critique from Raymond, he sent me a one-line e-mail quoting Leo Tolstoy:

"'It is amazing,' wrote Tolstoy, 'how complete is the delusion that beauty is goodness.'"

For Wall, goodness is making connections, which has a beauty all its own.

Perl glues more than just the world of C programmers to the world of Unix shells, or the digital severity of the computing world to the inchoate confusion of human reality. Perl also bridges the gap separating the two extremes of the free software world -- the commercial pragmatists represented most volubly by Raymond, who espouse free software principles because they produce the best software, and the radical moralists led by the Free Software Foundation's Richard Stallman, who believe in free software as an absolute good.

Those who wish to modify and change Perl can choose to do so under the restrictions imposed by two different licenses. They can operate under the GPL -- the extremely strict license created by Stallman's FSF that is aimed at ensuring free software always remains free. Or they can choose Wall's own "artistic license" -- a more forgiving set of rules that means, Wall says, that "essentially you can do anything you want with Perl as long as you leave me artistic control over it."

"It was enough to sneak it by the FSF without raising their hackles," says Wall, "but in actual fact the artistic license is closer to my feelings on the subject. I want Perl to be used."

And it is. Perl opens up possibilities in seeming dead-end alleys -- one major reason why so many Perl hackers, when asked why they love Perl, simply reply that it's fun.

I asked Larry Wall why he thought Perl was fun.

"It lets you be creative," he says. "It gives you choices. Your typical computer language is like a train, it will take you anywhere as long as there are tracks there already. Perl is more like an off-road vehicle that will get you where you are going, even if there is no official way to get there."

When you can facilitate that kind of path-finding, you reap what you sow. Larry Wall, sitting in his living room in Mountain View, petting his cats, listening to the weird sounds ricocheting around his house, looks like a pretty happy guy. He's given something very valuable to the Web, and the Web won't let him forget it.

"Hardly a day goes by that somebody doesn't write me a message saying that I love your stuff," says Larry Wall. "'It changed my life, I used to hate programming -- now programming is fun again.'"