comp.lang.perl.modules dbi-users@fugue.com Is perl and DBI good enough? """""""""""""""""""""""""""" Is it realistic to use perl with DBI for a large project for a big company? Should we use java servlets CGI instead of perl CGI? The question is: Is there any other big projects out there that has been acomplished with perl and DBI? Is perl good for anything else than guestbooks? I love perl, but we must make a serious evaluation of the alternatives. We must have support for the components we use. The Perl Clinic do not seem to guarantee the quality of the modules they support. I would like references to other big companies that use large custom perl programs as a part of their core business. "Internet Movie Database" is a free internet service, but not a system that MUST work. / Jonas.Liljegren@infohwy.se --- I got your message from Tom C. I can't speak for the Perl Clinic, but I'm going to soon be establishing a Perl support service. I won't be able to guarantee the quality of software I'm not writing or maintaining, but you should know that even without the Clinic or other services, you can get really good support from the fellow users and developers of modules like DBI. And, yes, Perl is definitely industrial-strength. It's not perfect, but among Perl releases, Perl 5.004 and its maintenance releases are of excellent quality. Well, Java is also a fine language. But from what I've seen, Perl allows greater programmer productivity than Java, for reasons that are analogous to why Java allows greater productivity than C++. In my opinion, you should start with Perl and see if it works out. If you have specific problems, then you can consider changing horses. BTW, I can't overemphasize the importance of the "-w" (warning) flag and the "use strict" pragma in creating and maintaining large Perl programs. Please be sure to use "-w" and "use strict" from the start of your Perl project. / Chip Salzenberg chip@pobox.com --- It's quite realistic to use perl. While the current website isn't using DBI/DBD, and Connect software uses it's own api to read/write to Oracle rom perl, we are developing all of our backend support for Oracle using DBI/DBD. Our database is business-critical, if it doesn't work, the business fails. Java takes too long to load, while for your clients that may be okay, many of ours are just a little bit impatient. I don't blame Perl Clinic for not guaranting DBD modules, they didn't write them. / Micheal Mc Evoy Webmaster/DBA email: mcevoy@artville.com voice: 608.243.5927 --- I worked on a large Web project where we used Perl and SybPerl. Not everything was done in Perl because some technical decisions were made w/o proper consultation of the technical staff. So a combination of C (through a Web/database development tool) and Perl CGI programs were developemed. All the backend administration jobs were done with Perl. If things were to be redone, I would advocate Apache with Perl. With mod_perl one should be able to reduce the performance hits of conventional CGI execution. Do not know about Java (I think it is over-hyped language). Have yet to find a case where Java would a better tool than Perl. It seems many people are using Java beacuse of the hype, and not on sound technical reasoning. / Earl Hood ehood@hydra.acs.uci.edu --- I am not sure about real *big* projects for *big* companies, but Perl+DBI surely work for medium scale projects (such as commercial grade billing system for a large ISPs) and its stability in my experience seems to be greater than that of JDK/JRE. / Alex Novgorodov drdivano@mail.axon.ru --- While we only have a couple *big* programs, we do have many (over 10 in heavy use) perl based cgi programs that are used for day to day business and must work all the time. Most of our applications are for dynamic reports from our database but a few are read/write applications. Most of these apps are low transactions (1 use every couple of minutes). / David Tauzell dtauzel@uswest.com --- Here at Silicon Graphics/Cray Research (10k+ employees, $4B US run rate), we use perl5 DBI CGI scripts as a front end to our machine database, as well as to generate financial reports for execs. and quarter end numbers. It works very well, is very fast, and can be used from any platform anywhere in the world. The code for the entire project is somewhere around 400k lines. / Justin Banks Silicon Graphics, Eagan, Minnesota justinb@cray.com --- I have the same problem as you: we need to move very quickly to web based data base applications so that we can relieve the tremendous burden of distributing client applications to the desktop (which doesn't work at all for a data base application distributed to the world). But the immaturity of the what industry is offering is troubling. I understand your concern that DBI may lack some things but to believe the hype that JAVA can solve all of these problems today is dangerous, in my experience. So let's look to the data base vendors: Their sales pitch is to use their proprietary tools (very anti Web - locks you in to their software only) to give you ease of development/maintenance, better performance, and better support. With at least Oracle and Informix tools, I have found none of these to be true. So what are we to do? I think industry has some maturing to do, but in the mean time we have work to do and we don't want to put our eggs in the wrong basket. The one thing that I am sure of is that efforts like DBI and Netscape's Livewire (serverside, DBMS independent JAVAscript) say clearly to the data base vendors we can do this without tying our applications absolutely to a single DBMS - a very important message. BTW, what environment are you talking about when you say, "Should we use java servlets CGI instead of perl CGI?" Are you talking about JAVA applications running as a CGI program? Does anbody know anything about the maturity of Netscape's Livewire? Does it work? / kjones@pumice.larc.nasa.gov --- We use perl 99% of the time on our sites (www.Austin360.com, www.GoBig12.com) for large projects such as our movie listings (perl, mSQL), restaurant listings (perl, DBI/DBD, Informix), chat (perl), bulletin boards (perl, but about to be perl, DBI/DBD, ePerl, mod_perl) and countless other things. Perl is robust, scalable and reliable. With mod_perl and Apache I see no reason to use another language over perl for web development. / Jason Bodnar Cox Interactive Media - Austin jason@cimedia.com --- I haven't done much with Java servelets, but here is what I feel are some advantages/disadvantages. Servlet Advantages: Java is a "better" language if you want to do OO stuff. It also has some better development tools. Since your servlet is multithreaded it will be much easier to cache database connections and maintain state across client requests. You can do this with some special perl/web server programs like 'mod_perl' and FastCGI, but since both are multi-process and not multi-threaded it will be a lot more difficult to share data. Perl Advantages: Perl is quite easy to use (IMHO) and has a huge library of freely available libraries. Using a pure CGI environment gives you the benefits of completely stateless system. For example, if your web server crashes or is restarted you won't lose anything that a servlet might be storing in memory. Of course, this can be handled with servlets but would take extra work. Perl Disadvantages: While perl is quite fast I think that in an application with a high number of transactions you would start to see the penalty of starting up a perl process for each request (again, you could use mod_perl or FastCGI to help make up for this). Perl has extremely limited CORBA support. This only matters if you want to use CORBA. There aren't a lot of perl development tools (i.e. IDE's debuggers, etc...). Perl's OO facilities aren't that great. Servlet Disadvantages: I think that writing Java programs will take more time. Part of this is because it forces you to think about good OO design, but also because Perl's syntax was designed for quick development. Servlets are in use, but they are still a new technology. There are only a few servlet engines available. Hope this helps a bit. / David Tauzell dtauzel@uswest.com --- the problem with CGI is that it causes the web server to fork each time it executes a script. This is an incredible amount of overhead, especially for the newer threaded servers. If you can use a servlet that runs as part of the web daemons then you will see a dramatic increase in performance, for instance Netscape's livewire will give you a 5-10x performance increase over straight CGI. Of course, you then have to worry about memory leaks .. but nothing's free :) / Michael Karafotis mkarafot@colltech.com --- I'm currently working for DataCash Ltd, in the UK. We're not huge. But Perl is central to our business model. We write software that takes in transactions originating on web servers and turns them into transactions in the protocol used by the UK clearing banks. Financial middleware, in other words. We're the only company in the UK that's allowed to clear American Express fund transactions from the internet. We already support four of the six largest public lenders in the UK, with the others in the pipeline; our customer base includes British Telecom, IBM, and others. We've been profiled in Computing, and we're growing. I expect that by this time next year our (Perl/DBI) based servers will be handling financial transactions in the sub-billion pound range; by the year after, well, your guess is as good as mine. I repeat: Perl is central to our strategy. And our perl code base is mission- critical in the sense that without it we don't have a business, and if it doesn't work properly, people will lose LOTS of money. Does this answer your question? / Charlie Stross charlie@antipope.org --- I did a large web site in perl, http://flirt.com I am now completing 1.1 of GoodTimes in java, http://www.opendoors.com/gt Of the two, you will find the reuseability and componentization far greater in Java. Code gets copied around a lot in perl making for maintenance nightmares but allowing simple jobs to be completed very quickly. If this is a long term project, with predictable development support, I would want to do it in Java using persistent Java servlets, not perl cgi. / Lane W. Sharman lane@san.rr.com --- > Should we use java servlets CGI instead of perl CGI? Up to you. If you go with Perl/DBI, look at Apache with mod_perl instead of CGI. It will be fast as hell. Our entire web site runs via Perl CGI/mod_perl, with DBI talking to an Oracle backend. http://www.arttoday.com/ The members-only site contains all of our intense Perl code. If you desire, I can write a small summary of our evaluation of Apache/Perl against Oracle WebServer, NSAPI, ISAPI (shudder), etc. If you want a free demo membership to ArtToday, reply and I'll fix you up. Then you can see what we're doing day in and day out. I depend on Perl not just for DB access and Web page generation, but for scripts that reconcile our financial data, authorize and capture credit purchases, manage encrypted credit card data, expire memberships, generate email form letters, page sysdamins when hardware fails, replace traditional client-server apps in-house, etc. etc. etc. In short, Perl has carried the load every time, and has saved us thousands of hours of development and maintenance. We make money every day with Perl. Take it very seriously. / Robin Thomas robint@zedcor.com --- My company is in charge of applications for the healthcare industry, the collection industry, and even for the government. Our systems *can not go down*. 99.9% of the development that we do uses Perl. It has proven to be scalable, modular, and the development time is amazingly fast. / Charlie Stross charlie@antipope.org --- um.... The Human Genome Project big enough? I think amazon.com runs on it. lots of things do. I use it for managing regulatory data in the pharmaceutical industry myself. > The Perl Clinic do not seem to guarantee the quality of the modules they support. Have you ever actually talked to tech support at Oracle, Microsoft, Netscape or any other company? None of them "guarantee" that their software will do anything at all and their tech support is usually clueless. I think 3M is currently about the 8th largest company in the US. Though they don't 'officially' support perl/DBI or even Oracle for that matter (corporate likes Sybase) / Gary Rich 3M Pharmaceuticals garyrich@smartlink.net --- We use Perl to insert data into production databases. We are a 400+ bed hospital and use C and Perl interfaces to transfer patient info. between new and Legacy systems. On it's way between these hosts we store this info. in a database, so that we can then redistribute it to more than one receiveing hosts. It works out to be a massive amount of 'Mission Critical' data. We like to use Perl because of the quick development turnaround, even for experienced C coders. We have had several of the Perl versions of these interface programs in productions for a few years with absolutely no problems whatsoever, other than a serious memory leak. Because of this one problem, we are backing away from its use, for large scale jobs. One program will grow from 8k to 60+ megabytes, in about a week, before the system shuts it down for lack of resources. We haven't gotten any response from the DBD/DBI group about this, and don't really expect to. Perl is most definitely an 'As Is' product. Then there is the continual hassle of upgrading. If you want any kind of response from the developers you must be prepared to upgrade first, and right now, we can't even go to the next level, because the latest version of DBI requires threads, which you have to obtain and install yourself. The developers stress that there are companies that will maintain your Perl for you, but I don't think that that is any kind of guarantee at all. Where business is concerned, you need to try to stick to the tried-and-true. One day the big vendors, like HP, IBM and DEC, will ship and support what they think is a stable version of Perl. Then you have a number to call where they'll have to give you some kind of direct attention. In the meantime, the developers kind-of expect you to be real Perl Savy, and fix your own problems as you go along.... / James Burn UMMS ITG jburn@itg.ummc.ab.umd.edu --- How does a service critical system processing over 400MB of call record data per day (that's almost 6 million call records), from hundreds of telephone exchanges, into a 60GB database sound? > The Perl Clinic do not seem to guarantee the quality of the modules they support. True. On the other hand _I_, as the creator and founder member of The Perl Clinic, do the DBI related support work. The system I mentioned above was developed by me and my company for British Telecom. You've probably heard of them. / Tim Bunce Tim.Bunce@ig.co.uk --- Read your licensing agreements some time--almost *nobody* guarantees anything, let alone code quality. (I'd hold Windows 95 out as an example, but that's a cheap shot) Check out the disclaimer that Oracle sticks on their software if you like. I've honestly come across very few pieces of software that are more reliable than perl and DBI. Nothing from Oracle falls into that category, that's for sure. And, of course, when perl fails you've got both the source and a large group of other people with the source who can help you quickly. Try that from Sun or Oracle... / Dan Sugalski SysAdmin Oregon Univeristy System sugalskd@ous.edu --- Oh yes. We at SegaSoft use perl/dbi for over 200 cgi's, a lot running as FastCGI demons. These cgi's although they are only 30-100 lines long each, they use about 16,000 lines of perl libraries that do the actual talking to the database. We have over 150,000 customers accounting for far too many cgi invocations than I want to think about. Granted, we are not a big company, but we place big demands on our web servers and what they have to do for us. / Scanner scanner@segasoft.com --- Perl is good, as we all say; but there's a downside as with everything. There have been cases in which data were lost due to memory running out, typically when doing global changes to multiple files. But that's just that. / Jie Gao jgao@csu.EDU.AU --- > I've honestly come across very few pieces of software that are more > reliable than perl and DBI. Nothing from Oracle falls into that category, > that's for sure. And, of course, when perl fails you've got both the source > and a large group of other people with the source who can help you quickly. > Try that from Sun or Oracle... We'll sell you the Solaris source if you really think you want it :-) / Alan Burlison Uk Performance Centre, Sun Microsystems alan.burlison@UK.Sun.COM --- Well over 50% of CGI on the Web is done in Perl. I went to Selena Sol's web site (she's written some perl CGI solutions) and found the following organizations using her stuff: (these are only the biggest) Brown University, Mindspring, US Naval Institute, Lucent Technologies, Novell, Electronic Frontier Foundation, Microtech Computers, Microline Software, Ellsworth AFB, British Columbia Ministry of Health, University of Iowa, Computer Review magazine, Greyhound Canada, Century 21 Realty, Boeing, SAS, Budget Inn, USDA, Howard Johnson, Northwestern University, Coldwell Banker, and National Center for Human Genome Research. They were using Perl for everything from database access to web shopping, including secure credit card sales. / Brett Slocum slocum@io.com --- Speaking of someone developing in Perl with DBI for a large project for a large company, yes, yes, and yes. Here's my experience. I have found Perl to be the most stable, error free development environment I have used. In 9 months of working heavily in Perl, I have found 1 bug in Perl. In the same time frame, I have found 12 bugs (some very serious) in Solaris 2.4, 4 bugs in Solaris 2.5.1, and too many to count in the Sun C++ compiler (the 4.1 compiler). The only software that rivals Perl in quality in my book is Linux (I have found 2 bugs in 3 years). As for DBI, I have found several "features", but no real bugs, yet. My biggest complaints about Perl with DBI would be the state of the DBI documentation lagging the implementation, and the lack of support for complex data types in the DBI interface. As for support, we have Sun "gold" support (7x24 hotlines, pagers, and on-site Sun support staff). The quickest turn around for a Sun bug fix I've seen was 2 weeks (and that had to be escalated to the president of AT&T Labs to get that fast a turn around). Of the three Perl bug reports I've entered, 2 were bogus (i.e. not Perl's fault), and I was notified of that fact within hours of reporting the "bug". The one "real" bug was minor, and I'm still waiting (I expect it'll be in the next release). Again the only rival is Linux, for which I had fixes for all 3 bugs within 1 day of reporting them. To those providing Perl and DBI/DBD development and support, thanks for your efforts! / Michael P Lindner mpl@wnmail.wndev.att.com --- You didn't define big, but we use Perl/DBI to load user input data into Informix, then again to run all the reports on the data which get sold. I just loaded about 15000 records yesterday, and I'm about to go run about 125 reports on that data. Don't know if that's big or not. :-) / Kevin Brannen kevin@inetspace.com --- Commercial support is way way over rated. Let's get past the myths and deal with the reality that most of the big s/w vendors have obtuse mechanisms for escalating support cases. Truth is, I find the company's engineers hanging out on USENET are far more helpful and timely than the formal support mechanisms. Free s/w such as Perl, Apache, DBI, FreeBSD and so on have usually have comparatively excellent support via public channels for the same reason: the folks who code the guts of these tools hang out on public forums such as these mailing lists and USENET. Admittedly, comp.lang.perl.misc sorely needs to spin off a comp.lang.perl.cgi so the guru's can get back to participating and stop bogging their attention down with flaming newbies for failing to post to comp.infosystems.www.authoring.cgi but that's another melodrama. In answer to your question, we have a mod_perl enabled server installation running on FreeBSD that so far happily supports 3 million requests a day -- this is an ad banner server, so you can imagine how mission critical it is. The next app we're launching makes extensive use of DBI, mod_perl and other tools that lack commerical support but I have a good deal of faith in the developers of these technologies -- more so than the support channels available from Microsoft, IBM/Lotus, etc. / Ian Kallen ian@gamespot.com --- The reliability of any big project is dependent on the programmers mainly. Perl is more than stable enough to trust it. Whatever your programmers come up with might not be. The same question would arise for C, or really any other language. Which guarantees do you have that a certain C compiler doesn't have a bug which might break your programs at some odd point? Will you also ask this same question wherever you go to get information about the compiler you will evaluate? / Martien Verbruggen Webmaster www.tradingpost.com.au --- We use perl and DBI to automate dataloads into the data warehouse at the bank I work at (you may have heard of us, on NYSE our symbol os CMB). I would say the perl and the database interfaces are "ready for prime time" / Kent Perrier kperrier@neosoft.com --- I would go so far as to say there are very few projects, even large ones, that should NOT be done in Perl. Granted, you have to use the right tool for the right job, and Perl is not always the right tool, but I believe it is the right tool for most common programming jobs -- especially cgi/web-style jobs. I'm a consultant and one of my clients is the Texas Education Agency (http://www.tea.state.tx.us). The project I'm implementing is a statewide education directory (Askted - http://Askted.tea.state.tx.us). The entire project resides on an IBM J40 RS/6000 - 8 PowerPC 604 processors 1 1/2 gig physical memory. Our project uses only a small portion of that computing power of course (Perl's not that big a hog!). Anyway, all the program logic is in Perl 5, mostly OO-oriented. The web server is Apache and we're using mod_perl for performance reasons. Our normalized database is Sybase and consists of around 100 tables. Data in the database is things like schools in Texas, districts, principals, addresses, etc. Additionally, we export data out to several other systems including an old mainframe and recent java-based systems. You can only access the reporting side of the project (which, by the way, is NOT mod_perl yet) through the web. Depending on your link and what report you run it might seem slow (some of the reports give you 100K plus of data). On the internal network things are snappy. There is a large admin side with many screens for adding/updating/deleting data that you cannot access without a username/pw (which I can't give out). Depending on your access level on the admin side you may or may not see certain options. It's actually pretty complicated. I have to say that doing this project in C, C++, or even Java would have been a pain. With Perl, rapid application development goes to a new level. Additionally, changes (and we've had a few) are usually pretty easy to implement. The number of available perl modules out there under CPAN, the quality of the database interfaces & other modules, and the power of Perl's syntax (string-handling, etc.), have really been a major plus. Several other projects are going on here -- one using traditional VB with ActiveX components, etc. a few using NetDynamics java environment, and one using a homegrown java environment & java webserver. Our Perl project easily holds it's own in terms of complexity, flexibility, and speed (under mod_perl). Of course, Perl does run slower than compiled C, so critical portions may need to be done in C, but so far we haven't had this problem. In fact, the time you save by using Perl in the first place often means you have the capability to go back over your code and optimize it -- which, depending on how well you wrote it the first time, might be a much bigger increase in speed than moving to a compiled language. We really only touch the surface of the mod_perl/Apache environment. We basically use it to pin programs in memory to avoid interpreter startup overhead and to cache in system memory some smaller lookup tables from the datbase, but it has a lot of integration into the Apache web server environment and makes a lot of neat things possible (dynamically configure your web server using Perl for example). At first, Perl syntax can look unmaintainable, but it's really not -- it's certainly no worse than C's long series of pointers, casting, etc. If you're doing a large project I highly recommend using the object oriented stuff in Perl5 (which is basically optional to use). It help keep things clean. There are some tricks in Perl that you need to know so have someone who knows the language or spend a good bit reading the books -- things like 'use strict;', powerful regexs, autoload function, etc. can be very useful. Finally, you cannot beat the support from the comp.lang.perl* newgroups. / Dylan Tynan aixgod@ix.netcom.com --- Interpreters (such as Perl) allow easier debugging than compilers (that's one of the main advantages of them). As about extending, Java seems to have a little advantage over Perl, since it has more strict syntax, but it's a minor advantage in my opinion. If you need to get people to write in consistent manner, then you need to design a proper policy. Bondage and discipline language cannot replace a good policy. The advantage of Java over Perl is that Perl is absolutely not suited for developing complex user interfaces, but who says that you can't use Java for GUI and Perl for server backend together. Also, the fact of using Java may make your bosses happier :) / Alex.Novgorodov drdivano@ddcorp.spb.su --- > Is it realistic to use perl with DBI for a large project for a big > company? Almost assuredly, which doesn't release you from the responsibility of sizing your loads and configuring your application accordingly. > Should we use java servlets CGI instead of perl CGI? So you're application is CGI, eh? Sure you could, however, Java is slow enough without starting the VM everytime a Java program needs to execute. Is the Java VM embedded in the web server? Great; but you can get the same effect, though probably better, using Apache with mod_perl. By the way, although Java has a cloud of hype surrounding it, I consider it one of the best ideas to ever actually make it to market. It's unfortunate that Java has been positioned in any way as a 'competitor' to Perl, since they are so different. Java needs to compete against C++, not Perl. At least Larry seems to realize that Java and Perl could make a perfect complimentary pair of languages. > The question is: Is there any other big projects out there that has > been acomplished with perl and DBI? Is perl good for anything else > than guestbooks? Well, I don't know how 'big' we are, but we are an FDA-regulated establishment and the reliability and safety of our software is not just critical to us, but the health and welfare of the patients we serve. That said, we have no compunctions using perl, DBI and DBD::Oracle to write all the extensions to our main database application. > I love perl, but we must make a serious evaluation of the > alternatives. We must have support for the components we use. The > Perl Clinic do not seem to guarantee the quality of the modules they > support. And if you go to a commercial vendor, you will notice that if they even do support their software in the way you describe (normally, you just get a page full of disclaimers), the supported behavior is narrowly defined. In the real world, even from a commercial vendor, you will often run into the "That's not a bug" problem or "We don't debug code so don't even show us your program" problem. On the other hand, by undertaking a large project in Perl, you extend the popularity, image and usefulness of a language of which many people are enthusiasts; you will become part of a team of people, some expert, which resources you can depend on tapping. It will not just be you and your programmers facing the project but, in a larger sense, the Perl community. I have found this spirit of teamwork to be a difficult-to-quantify but invaluable and critical part of successful Perl projects. I hope this helps you in your decision. / Jeremy Brinkley System Administrator Stanford Blood Center jeremy@wishbone.stanford.edu ---