Perl 6 and the Future

part of Perl for the Web

Perl is a language that has undergone many changes. (It has grown a great deal.) As Perl grew, the core constructs of each new version were updated to help serve the needs of the latest generation of Perl programmers. Perl 5, for instance, marked a turning point in Perl development by providing a framework for writing standardized extension modules in Perl. Perl 5 also added the capability to access these modules using an object-oriented idiom instead of importing functions. This enabled module programmers to concentrate on writing useful functions without worrying about naming conventions or overlap with other modules. These language changes paved the way for a profusion of Perl modules that customized the language to suit any number of tasks, including the design of Web applications.

Perl 6 provides an opportunity to remake the language again to suit the needs of current and future Perl programmers. Lessons have been learned from the way programmers used and extended Perl 5. There now are more programmers with intimate Perl knowledge than when Perl 5 was being designed. The core language designers, Larry Wall included, have learned a great deal from new languages, such as Java, that didn't exist when Perl 5 was designed. As a result, the change in Perl will be radical to incorporate these new insights; Perl will be rewritten from scratch for Perl 6. Support for new technologies such as Unicode will be built into the language from the ground up instead of tacked on, and special attention will be given to making the internals of Perl more accessible to programmers who want to seamlessly extend the language.

Perl 5 to Perl 6 Translation

One exciting aspect of Perl 6 development is the emphasis on an easy transition between Perl 5 and Perl 6. Many of the programmers working on Perl 6 remember the difficulties that were caused by the switch from Perl 4 style to Perl 5 style. The need to retain backward compatibility with Perl 4 programs made it difficult to add new features to Perl 5. At the same time, the few changes made were enough to break many existing Perl packages. With these lessons in mind, it was decided that backward compatibility with Perl 5 was important, but that it shouldn't impede the progress of Perl 6 toward bigger and better things.

Fortunately, an ingenious solution was found to the problem of updating Perl 5 programs to be compatible with Perl 6 syntax. Instead of trying to implement Perl 5 syntax as a subset of Perl 6, the decision was made to incorporate the automated translation of Perl 5 programs into Perl 6 style (or vice versa, if necessary). In fact, one test of completeness will be the ability to use all CPAN modules with Perl 6 by simply running them through the translator. This amazing idea frees Perl 6 designers to make even radical changes to the structure of the Perl language without worrying about leaving older Perl programs and modules behind. Of course, another principle adopted for Perl 6 design is "Perl should stay Perl," so the changes aren't likely to be so radical that Perl won't be recognizable.

The Perl Compiler

The essence of a Perl 5 to Perl 6 translator comes from the way in which Perl compiles source code into an executable program. When a Perl program is run, the Perl compiler loads the source code of the program and compiles it into bytecode that the Perl runtime can execute. This compiled bytecode normally is discarded after the program exits, unlike the bytecode output of a Java compiler. However, the bytecode itself serves a similar purpose–it's a cross-platform compiled representation of the Perl program that has been parsed, optimized, and prepared for the Perl runtime.

Perl 5.6 offers a suite of compiler backend modules, collected under the B module, that can be used to control the behavior of the Perl compiler. Using the B::Bytecode module through the O interface module, Perl programs can be compiled into the bytecode format without executing them. For instance, a Perl program called generate_table.pl could be precompiled into bytecode using the following Perl command line:

Listing 16.

perl -MO=Bytecode generate_table.pl

The -M switch tells the Perl executable to load the O interface module with the Bytecode backend module to process the generate_table.pl program. Additional options can be specified to include additional modules when compiling or controlling the effects of compilation. The resulting bytecode file is sent to standard output, so it then could be stored for later use, either in memory or on disk. See the "Compiling Stand-Alone Programs" section later in this chapter for examples of how bytecode could be used to improve application performance.

The Perl 6 Decompiler

Unlike the compiled versions of programs in most languages, Perl programs can be decompiled from the bytecode back into Perl source code. In fact, this process can be modified to produce different versions of source code that would produce the same bytecode result. In Perl 5.6, this is accomplished using the B::Deparse backend module, which can produce valid Perl source code of various types from a Perl program. B::Deparse most commonly is used to check for unwanted behaviors in the way Perl interprets the original source code by making it produce an annotated or expanded version of the code after parsing. For Perl 6, though, additional options could be added to cause B::Deparse to emit Perl 6 code from a Perl 5 source, or vice versa. The command-line options for such a conversion might be similar to the following:

Listing 16.

perl -MO=Deparse,-ogenerate_table.pl6 generate_table.pl5

It still remains to be seen how the translation will be implemented, whether as a Perl 5 module for conversion to Perl 6 or as a Perl 6 module with backward-compatibility built in solely for the translation process. Additional features will probably need to be added to deal with formatting–including indentation and idiomatic style–and comments, which are currently stripped out of the source code during optimization. Other options will be implemented specifically to handle the translation of modules, including those modules that have a C library component.

Language-Independent Perl

Another exciting aspect of the Perl 6 compiler is the possibility of compiling code from other languages into Perl binaries. Perl 5 enables this in a limited sense; the XS extension mechanism can be used to incorporate C libraries into Perl modules, and the Java-Perl Lingo (JPL) was developed to incorporate Java programs into a Perl framework. However, both mechanisms are so difficult to use that they are reserved for modules that desperately need them. In addition, neither mechanism provides a framework for other languages, so each new language supported has to be incorporated separately. With Perl 6, the idea is to elevate other languages to first-class citizens in the Perl compiler. A Perl program or module could be written in Perl, but it also could be written in Java, Python, C, or Ruby.

This might seem like a frivolous use of a compiled environment. After all, most languages already have some form of compiler or interpreter, so there should be little need to compile programs in those languages into Perl. However, this assumes that each program is being developed in a vacuum, with no need to run programs from multiple languages in the same shared environment. With Web programming in particular, this couldn't be further from reality. In most Web environments, applications can come from a wide variety of sources, spread both over time and over multiple organizations. You can't assume that all these groups will use the same language or environment, so it's sometimes necessary to collaborate between groups that use incompatible tools. If Perl has the capability to run the original code written by each group, no matter what the language, it has a much greater chance of being the platform chosen to implement a combined solution.

Java to Perl

One language that is bound to be important to Perl developers is Java. Despite the excellent track record of Perl in implementing Web applications, many application designers still are more comfortable learning to program in Java than in Perl. Regardless of their motivations, programmers who prefer Java or who have existing code written in Java might end up on a Web application project developed in a Perl environment. Rather than discard their existing code and experience, it would be good to allow them to program in the Perl environment using the Java code they're used to. Even better would be to allow them to program parts of a program in Java and parts in Perl so that they become used to both languages over time.

In addition, Java does have the advantage of a vigorous corporate following. Many projects coming out of large corporations, such as IBM, are being developed in Java first–or Java only–with the idea that the Java code will serve as a reference implementation for others. Instead of tossing the reference code aside while reimplementing the project entirely in Perl, it would be good to allow these projects to be incorporated as Perl modules in their native form. Even better would be to allow Perl programmers to optimize parts of a Java module in Perl as the project progresses. This could serve to improve the availability and completeness of Perl modules in crucial areas such as Extensible Markup Language (XML) processing and legacy system interaction.

The Inline Module

The beginnings of a standardized approach to incorporating other languages into Perl comes from the Inline module. New to Perl 5.6 and still experimental, the Inline module enables code from other languages to be used directly in a Perl program as though it were native. The module creates a framework similar to the DBI module, in which additional languages can be added as drivers while the API for using them stays constant. A program using the Inline module can specify a language (or languages) to include inline. It then can write and call subroutines from the additional languages as though they were Perl subroutines. Code segments in other languages aren't included in the main Perl block, but they can be included as DATA sections in the same file as the Perl program. Alternately, source code in other languages can be imported from other files into the Perl program using the Inline module and then accessed as though the subroutines were imported Perl subroutines.

A crucial purpose to which this kind of language-neutral implementation can be put is the creation of Perl modules in other languages. Sometimes, Perl isn't able to benefit from work carried out initially in another language. In addition, the Perl modules created as derivative works might be missing some of the features of the original libraries. For instance, Perl support for the XML Document Object Model (DOM) lagged behind other languages, such as C and Java. Perl's XML::DOM module stagnated for nearly a year while similar C libraries added new DOM features and improved performance. If the process of adapting C or Java libraries for use as Perl modules were streamlined, Perl could benefit from work done in any language as soon as it became available.

In Perl 5, the Inline module is implemented by automating the process of creating XS modules, which then are imported into the namespace of the main Perl program. However, early information about the direction of Perl 6 hints that this kind of multilanguage support will be built in at the compiler level, so implementing Inline language drivers will become considerably easier in Perl 6. Support for code in C, C++, and Python already is incorporated into the Perl 5.6 version of Inline. Java, Ruby, and other new languages are expected to be added as drivers before Perl 6 is released. Perl 6, in turn, probably will bring about a marked increase in the number of languages supported as well as the number of Inline-based modules created, just as Perl 5 brought about the rapid growth of Perl modules.

Perl 6 Performance Improvements

One area in which Perl 6 is hoped to surpass Perl 5 is performance. Perl performance is usually the subject of fierce debates, with C programmers claiming that Perl programs are too slow overall and Perl programmers claiming that Perl programs are capable of being as fast as C programs. (Perl programmers also add that the development time necessary for a C program generally makes up for any performance loss in a Perl program.) Regardless of the current state of Perl performance, all sides would agree that making Perl faster overall would be a good thing.

As a result, the development of Perl 6 is being carried out with an eye toward improving both the perceived performance and the benchmarked performance of Perl programs. The internals of Perl are undergoing great scrutiny, with an eye toward reducing the memory footprint of the Perl runtime by moving unessential functions from the core into loadable modules. The implementation of the Perl compiler also is being rethought from the standpoint of optimizing even more Perl functions based on the way in which programmers are likely to implement them. In addition, aspects of the Perl language from object-oriented syntax to the behavior of unary operators are being updated to make both development and execution of Perl programs faster.

Compiling Stand-Alone Programs

One area in which the perceived performance of Perl programs can be improved is the time taken for compiling the program and loading the Perl runtime. As mentioned throughout this book, there are many situations in which compiling a Perl program every time it is executed can cause a considerable performance drain. If it were possible to keep Perl programs in precompiled states and use the precompiled versions when available, much of the time spent compiling programs could be saved. This would improve performance not only for Web applications, but also for complex stand-alone applications that currently take seconds or minutes to compile before executing.

As mentioned, Perl programs are compiled to a platform-independent bytecode before being executed by the Perl runtime. To reduce the need for compiling Perl programs at runtime, it would be possible to keep compiled bytecode either in memory or on disk and use the precompiled copy rather than the source copy to run a program. The B::Bytecode module currently provides a way to save bytecode and the B module enables it to be imported into a Perl program at runtime. For Perl 6, both systems could be streamlined and offered as universal Perl options, which would automate the process to some extent. Web environments would benefit from such a system by using compiled bytecode versions of Perl programs instead of subroutine-based execution. Bytecode execution would provide a more pristine environment to Perl programs, which otherwise would have difficulty running in a persistent environment. It would do this while providing most of the benefits of persistence.

Perl to C Compiling

Aside from bytecode, it's also possible to compile Perl programs into a system-specific language such as C. This enables the program to be compiled into a binary executable using a C compiler. Although this won't necessarily result in a program that runs faster than the equivalent Perl program overall, it can dramatically reduce the perceived performance by removing the compile step at runtime. In addition, compiled binaries are sometimes more compact and easier to distribute than the equivalent Perl programs. Compiling C programs would enable developers to control which version of Perl is used by a program, as well as the modules made available to it.

Compiling a Perl program to C code is currently made possible by the B::C module. An additional module, B::CC, compiles Perl programs into optimized C code with more restrictions. Both modules are considered experimental in Perl 5.6. Neither module generates a program that would be familiar to C coders translating a Perl program by hand. Rather, the program is closer to a representation of the process carried out by the Perl compiler and runtime when executing Perl bytecode. However it is implemented, the end result of the compiled code is a program that should work just as the original Perl program would have. Using the current modules, a Perl program named generate_table.pl could be compiled into C source code using the following command-line options:

Listing 16.

perl -MO=CC generate_table.pl > generate_table.c

As with the Bytecode and Deparse backend modules, the O interface module is used to invoke the CC backend module. Options can be provided to the CC module to affect the way the C code is optimized as well as how it is likely to be compiled. Other options can be specified to control which Perl modules are included when creating C code.

Historically, compiling a Perl program into C code has been a difficult prospect. The code produced by the C and CC backend module tended to be overly complex because it needed to reimplement much of the processing that would ordinarily happen inside the Perl runtime. The complexity of programs also tends to increase rapidly when resource-intensive modules such as Tk or XML::DOM are incorporated. The compiled code also has a few less-than-optimized side effects due to the Perl 5 implementation of the Perl compiler. For one, a full copy of the Perl runtime library is included in every compiled program. This means that the memory footprint of a compiled Perl program is likely to be as large or larger than the same program run through the Perl runtime. These kinds of problems are at the front of developers' minds as they create Perl 6, so it's likely that solutions will be implemented to enable fast executables to be generated from standard Perl programs.

Sidebar: Larry Wall and Damian Conway

By and large, the design of Perl has come from the mind of one person: Larry Wall. Larry wrote the first versions of Perl, and he has served as the chief language designer and spiritual leader for Perl design for each subsequent version. It's not too much to say that Perl would not be Perl without Larry; his intelligence, wit, and sheer practicality are embedded into the structure of Perl as deeply as the semicolon. (Sorry, Larry.) The Perl community also has benefited from his example. More than any computer technology I've encountered, Perl has a community of recognizable faces–accessible ones, even! Specific names are identified with each Perl project, and the result is an environment that encourages personal responsibility for code and pride in a job well done. Again, without Larry's example, I'm not sure that Perl would have such a genuinely competent community surrounding it.

More recently, another recognizable face has come along with a passion for Perl that rivals even Larry's. Damian Conway is a Perl programmer, lecturer, and author extraordinaire who is widely known for doing the work of a dozen eccentric geniuses. He has written truly amazing Perl modules such as Coy, which creates poetic error messages, and Lingua::Romana::Perligata, which enables Perl programs to be written using the position-independent syntax of Latin. His pursuits aren't frivolous, though. He has a knowledge of Perl to rival Larry himself, and each new module is an exercise in stretching the limits of Perl programming. His creations are so impressive that he's been barred from winning any further Larry Wall awards at the Perl Conference in exchange for having the Best Technical Paper category renamed the Damian Conway award. Every aspect of Perl touched by Damian seems to sparkle afterward, and it's work such as his that renews my faith that Perl development will be extraordinary for years to come.

Summary

The future of Perl looks bright. Perl 6 promises a complete rewrite of the Perl language and Perl internals, with a specific focus on improving the performance and usability of Perl. Radical changes can be made to Perl 6 because of a decision to implement backward compatibility by translating Perl 5 programs into Perl 6 automatically. This kind of translation also provides a promising framework for incorporating code from other languages directly into Perl programs. Support for these languages is being added experimentally by the Inline module, but Perl 6 should provide more seamless integration possibilities. Updates to Perl internals also will be a boon to efforts surrounding compilation of Perl programs as bytecode or executable binaries, both of which will find many uses in the years to come.