Open Source Best Practices Experiences from within the Apache Software Foundation Brad Nicholes Senior Software Engineer, Novell Inc. Member, Apache Software Foundation [email protected] Agenda • • Introduction to Open Source Open Source Projects – • Rules of Development – • • 2 Apache Software Foundation Getting Involved Best Practices – Patching – Development – Distribution – Communication Development Process © Novell Inc. All rights reserved Open Source Introduction “Open Source” is the concept of taking an idea, opening it up to public discussion and developing it into something that can have a broad benefit to an entire community. In its purest sense, it is an entirely free and open process whose contributions are made on a voluntary basis and whose result benefits all. 3 © Novell Inc. All rights reserved Open Source Introduction • 4 What is “Open Source”? – Free software that ships with the source code – Movement that has had a huge impact on the way software is developed – Collaborative software development process (even chaotic at times) – A driving factor in how companies do business whether they develop software or simply use it © Novell Inc. All rights reserved Open Source – Basic Idea “The basic idea behind open source is very simple: When programmers can read, redistribute, and modify the source code for a piece of software, the software evolves. People improve it, people adapt it, people fix bugs. And this can happen at a speed that, if one is used to the slow pace of conventional software development, seems astonishing.” OSI – http://www.opensource.org 5 © Novell Inc. All rights reserved Open Source Misconceptions • • 6 User point of view – A community of hackers with nothing else better to do – Only programmers can participate in the Open Source community – Open Source software is typically beta quality – It's free, I can do anything I want with the source code Corporate point of view – It’s free, we can’t make money with Open Source – The only good software is proprietary software – NIH (“Not Invented Here”) Syndrome © Novell Inc. All rights reserved Open Source Reality 7 • All Open Source projects have an owner • Anyone can participate in Open Source regardless of their skill level or technical knowledge • Volunteers may actually be paid by a third party to participate in an Open Source project • Results can be freely used by the entire community • Results may be used to generate revenue directly or indirectly © Novell Inc. All rights reserved Open Source Legal Issues • There are legal and business issues involved with participating in Open Source Projects: – 8 Avoid contamination > An engineer working on a given proprietary project may not be allowed to contribute to a specific OSS project > Importing GPL code into a proprietary project may force the project to be released under the GPL – Understand copyright issues – Committers to projects must do due diligence to resolve source code ownership – Comments and opinions made through your company's association ([email protected]) must be consistent with your company's policies and views © Novell Inc. All rights reserved Communities and Projects 9 • A community is an organization made up of ideas and concepts around a particular discipline, product, vertical market, technology, development methodology, etc. • A project is centered around the creation, development, and distribution of a software product. Projects are often associated with communities. © Novell Inc. All rights reserved What is an Open source Project? • Collaborative development environment – – – • License agreement – • All Open Source code is protected by some type of license agreement; the license defines the conditions under which the source code may be used and the obligations with respect to enhancements and derivative works Project management – – 10 Project contributors are often geographically dispersed and usually collaborate via email or IRC Email and email lists are the most popular means of communication. Thousands of emails are sent and received everyday discussing various projects Open Source projects are hosted at sites such as SourceForge.org, Savannah.gnu.org and Forge.Novell.com Most Open Source projects are managed by a single individual Larger projects such as Linux, Apache, OpenLDAP are managed by a group or organization with assigned roles such as “Release Manager” or “Sub-system Maintainer” © Novell Inc. All rights reserved The Apache Software Foundation What is the Apache Software Foundation • Membership based non-profit organization incorporated in the state of Delaware, USA • Provides organizational, legal and financial support for a broad range of open source software projects • Provides a hardware, communication and business infrastructure • All volunteer organization • Governed in accordance to bylaws – 12 http://www.apache.org/foundation/bylaws.html © Novell Inc. All rights reserved ASF History 13 • Founded in 1995 as the “Apache Group” • Incorporated in 1999 as the Apache Software Foundation • Purpose was to maintain and support the original HTTPD web server written by the NCSA • Became the #1 web server within a year after the organization of the Apache Group © Novell Inc. All rights reserved Apache Name “The name 'Apache' was chosen from respect for the Native American Indian tribe of Apache, wellknown for their superior skills in warfare strategy and their inexhaustible endurance.” “It also made a cute pun on “a patchy web server” -- a server made from a series of patches – but this was not its origin.” http://www.apache.org/foundation/how-it-works.html#history 14 © Novell Inc. All rights reserved Philosophy – “The Apache Way” 15 • Collaborative Software Development • Commercial-friendly Standard License • Consistently High Quality Software • Respectful, Honest, Technical-based Interaction • Faithful Implementation of Standards • Security as a Mandatory Feature © Novell Inc. All rights reserved Meritocracy 16 • Govern of Merit • The more you participate the more responsibility you merit • Easily able to weed out those that were just passing through from those that really have an interest • Allows for all levels of participation • The ASF only recognizes individuals rather than corporations © Novell Inc. All rights reserved ASF People • Member – 230+ Members – 13+ Emeriti Members Nominated by current members and elected on merit – Holds the right to vote on ASF related issues – All members are also committers – • • Committer – 1400+ Committers ... and growing – Given 'WRITE' rights to the source code repository – Assigned an “apache.org” mail address Contributor – – • 17 Contributes code and/or documentation in the form of patches Actively participates on the developer mailing lists User – Uses the Apache software – May provide feedback in the form of bug reports or feature suggestions © Novell Inc. All rights reserved Infrastructure • The ASF is a virtual entity that only exists on the Internet • The only physical existence of the ASF is the server hardware itself which provides: • 18 – Web Services (Sites and Wikis) – Source code repositories – Mailing list environment – Bug tracking – Distribution mirroring All services and hardware are managed by the Infrastructure Team which is also an ASF project © Novell Inc. All rights reserved Getting Involved My Involvement with the Apache Developer Community • • • Project Committer (since March 2001) – HTTPD (1.3, 2.0, 2.2) – APR – APR-Util Maintaining Apache 1.3 and 2.x for NetWare Writing/Maintaining Various Apache Modules – NetWare > – Cross Platform > • 20 Mod_LCGI, Mod_NDS, Mod_HDirs, Mod_RDirs, Mod_EDir, Mod_NW_SSL Mod_Auth_LDAP, Mod_Authnz_LDAP, Mod_LDAP, Mod_Authn_Alias Authentication and Authorization © Novell Inc. All rights reserved My Involvement with the Apache Developer Community 21 • Member of Apache Software Foundation (since May 2002) • Member of Various Project Management Committees • Conference Presentations & Sponsor – BrainShare (U.S. & Europe) – ApacheCon (U.S. & Europe) © Novell Inc. All rights reserved Becoming an Apache Committer • • 22 Making enough noise to get noticed – Asking questions – Posting patches – Doing the work that nobody else wants to Finding a Niche – Apache for NetWare maintainer – LDAP authentication – Authentication & Authorization © Novell Inc. All rights reserved Becoming an ASF Member • • 23 Member privileges – Nominate others for membership – Access to all foundation records – Access to Members', Board & Security mailing lists – Annual members' meeting Many opinions about what makes a good ASF member candidate – Active in multiple projects – Active in foundation affairs – Members must exercise their right to vote – Attend the annual members meeting © Novell Inc. All rights reserved Best Practices Rules of Open Source Development 1. Let the source be open 2. Release early, release often 3. Reward contribution with praise 4. Never “fork” a project and continue development behind closed doors 5. Adapt to the coding style or other standard practices outlined by the project 25 © Novell Inc. All rights reserved Rule #1: Let The Source Be Open • • • • • Have no secrets Make the code and the process that produces it, public Encourage third-party peer review Make sure that others can modify and redistribute the code freely Grow the developer, tester, documentation communities as large as possible – • 26 Encourage users and testers to become developers An efficient feedback channel from users/testers to developers is a major factor in producing high quality open source software © Novell Inc. All rights reserved Rule #2: Release Early, Release Often • • • • 27 Rapid release means quick and effective feedback When incremental releases are small, changing course to respond to feedback is easier Make sure that your first release demonstrates promise – It doesn't have to be fully functional – It must at least build and show elements of base functionality Don't make releases a momentous events – It is important to streamline your release process so that releases can be made often and painlessly – Resist thinking that the next release must be a polished jewel that can not ship until everything is perfect © Novell Inc. All rights reserved Making Source Code “Open” Source code • Have a clear idea of what problem you are trying to solve and how the project will be useful to the community • Resist the temptation to make the project “Code Complete” or “Perfect” before releasing it • 28 – A new large code base is difficult to review and understand – A design that is already set in stone leaves little room for innovation – Opportunities to join forces with similar projects are lost which could have resulted in rapid community growth and acceptance GO PUBLIC EARLY! © Novell Inc. All rights reserved Release Methods 1. Beta, Beta, Beta..., Release and Repeat – Avoid thinking that all must be perfect – During development, produce as many beta releases as necessary – Before a release, produce “Release Candidates” – Even if you are not ready to release a beta yet, make any and all source code available to the community 2. Stable Release Branch vs. Unstable Development Branch – Usually implemented by more mature projects – Allows maintainability of a stable product while fostering innovation and creativity for the next major release 3. All of the above... 29 © Novell Inc. All rights reserved Rule #3: Reward Contribution with Praise • If you can't give co-developers material rewards, give them psychological ones – • • 30 Being able to attach one's name to a well accepted piece of software is in itself a reward Most Open Source developers are motivated by two things – Increased usefulness of the software to them or to their employer – Honor, reputation and exposure that one can earn through open source software development Software creation is especially satisfying if a large audience welcomes it and finds it useful © Novell Inc. All rights reserved Rule #4: Never “Fork” a Project Behind Closed Doors 31 • Denies the Open Source community the opportunity to participate • Eliminates valuable feedback that would have increased the quality of the software • Eliminates development and testing resources that could have helped speed up the development process • Demonstrates that we really aren't Open Source community players • May violate the license agreement that governs the source code and all derivative works © Novell Inc. All rights reserved Avoid Forking A Project At All • • • • 32 Your goal should be to participate with and contribute your work back to the project If you are porting the source to a new platform, forking the project will require you to continually re-port the code Forking a project dramatically increases the probability of having to debug and fix issues that have already been resolved by the community Forking a project makes it much more difficult to have your changes accepted back by the project – Source code baseline may be very different – No relationship established with the project maintainers © Novell Inc. All rights reserved Rule #5: Adapt to Project Processes and Standards • 33 Use the coding standards and styles that are outlined by the project – Put aside your own style beliefs – If no coding standards are specified, use the existing source code as your guide – Same goes for documentation • The goal is to establish a relationship with the project, not to reform it • There is no problem with suggesting a change in process, just be prepared to accept the project's decision © Novell Inc. All rights reserved Best Practices When Working With Open Source Developers 34 • Good Patching Practice • Good Project and Archive Naming Practice • Good Development Practice • Good Distribution Making Practice • Good Communication Practice © Novell Inc. All rights reserved Good Patching Practice • • 35 Make sure your patch is easy to read and understand – Others need to review the patch and merge it, therefore a sloppy patch will likely be rejected – Sloppy patch creation and submission is usually a sign of sloppy code One fix, One patch – Don't submit multiple fixes or enhancement in a single patch file – If the fixes can be logically separated, submit separate patches with their own descriptions – The more complex the patch, the harder it is to review – One patch may span multiple source files but it is still one patch © Novell Inc. All rights reserved Good Patching Practice • • • 36 Send patches not files – A patch is a unified diff between the original and changed source – A patch can include changes to multiple files – Send an entire file only if your patch includes a new file Send patches against the current code base – Extremely difficult to determine your changes from changes made “nn” revisions ago – Your patch may actually be obsolete if based on old code Patching tools – Besides a compiler, “diff” and “patch” are the basic tools of Open Source development – CVS and SVN (Subversion) are popular revision control systems © Novell Inc. All rights reserved Good Patching Practice • 37 Send a minimal patch – The patch should only contain relevant code changes, documentation and comments – Use the unified diff format not the default > “diff -uNrp” is normally used to diff trees > “diff -up” for a single file – Don't mix code “style” changes with code “logic” changes – Include an explanation or description along with the patch submission of what your patch does or fixes and why – Don't be afraid to point out weaknesses © Novell Inc. All rights reserved Good Patching Practice • • 38 Don't forget the documentation – If your patch includes user-visible changes, make sure you also update the man-pages or any other documentation – Comment your code so that it is easier for others to review as well as to understand later – Make note of any user-visible changes or significant changes or features in the project's CHANGES log DON'T TAKE IT PERSONALLY IF YOUR PATCH IS REJECTED © Novell Inc. All rights reserved Good Development Practice • Don't rely on proprietary tools or utilities – The farther you stray from accepted Open Source tools, the less likely it is that your project or patches will be accepted – Keep the build procedure as simple as possible > • 39 Configure; make; make install Stick to standard libraries and API's that are widely available – Keep portability in mind – Usage of cross-platform libraries will make it easier for others to port your project to other operating systems – It is more likely that your code will be reviewed and accepted by the project if you keep with accepted standards and RFC's © Novell Inc. All rights reserved Good Development Practice • • 40 Keep #ifdef's to a minimum if possible – Stay away from platform specific code unless absolutely necessary – Stay away from compiler or linker specific options – Abstract away platform specific code if possible – Use common programming practices Test before you release – Provide your own test suite – Test or ask for testing results from as many platforms or platform variations as practical – Build with different compilers and test – Use testing tools where possible © Novell Inc. All rights reserved Good Distribution Build Practice • 41 Include various description and information files – README – new and important information – INSTALL – configuration and build instructions – CHANGES – significant version-by-version changes – COPYING/LICENSE – project license terms – FAQ – common questions and answers – HISTORY – project history – NEWS – high level list of enhancements or features © Novell Inc. All rights reserved Good Distribution Build Practice 42 • Make sure tarballs expand into a single new directory • Respect and follow standard file naming practices • Provide checksums so that others are able to verify origins of the tarball, RPM or other types of packages • Larger projects will take advantage of mirror sites to decrease the number of direct downloads and required bandwidth © Novell Inc. All rights reserved Good Communication Practice • Host your project on a well known Open Source hosting site (Forge, SourceForge, Savannah, etc.) • Provide various mailing lists that interested parties can subscribe to • 43 – DEV – developers' list – HELP – user help list – ANNOUNCE – announce new releases or significant changes – DOC – documentation development list Announce new releases or other significant events on the project's mailing lists as well as other relevant news groups or community lists © Novell Inc. All rights reserved Good Communication Practice - Email • 44 Because email is the prominent form of collaboration for OSS projects, good email habits are important – Use text mode email clients rather than HTML based. Plain text has the advantage of being easily quoted and referred to in replies – Retain minimal context when replying to an email by duplicating the referenced paragraph or quote with indented “>” symbols rather than including the entire message – Stick with short and clean statements rather than a “longwinded” email – Long attachments or binary data should be referenced on a web or ftp site for those that are interested © Novell Inc. All rights reserved Good Communication Practice - Email 45 – Vacation messages or bounces should never to directed to a public mailing list – Large communities tend to attract people who enjoy controversial, insulting or off-topic discussions. RESIST THE TEMPTATION TO PARTICIPATE – Project participates are expected to speak for the project rather than their company or personal interest – If a project provides several lists with different topics, try to comply with their policies and avoid posting off-topic messages – Avoid cross-posting messages to a large number of email lists © Novell Inc. All rights reserved Good Communication Practice • 46 Build a web-site that includes – Project goals, guidelines and standards – Quick link to all downloads and archives – HTML-ized version of all documentation – Links to source code control system, bug tracking and other online utilities – Information about how to participate – Project roadmap and/or schedule – Who's who and history © Novell Inc. All rights reserved Good Communication Practice • 47 Always stay positive and constructive – All communication should work towards moving the project forward – Suggest ways to make things better rather than different – Don't be afraid to point out a problem but be prepared to accept and correct your own problems – Use criticism or rejection as a learning tool for next time – Be prepared to accept the decisions of the project maintainers – DON'T flame the mailing list © Novell Inc. All rights reserved Project Development Cycle • Allow users and developers to download the source code • Accept changes and enhancements to the project • Determine when a release of the software is appropriate • Select a Release Manager • Tag a snapshot of the source code (optionally provide binaries) • Request that the community test the software on the various platforms and provide feedback • Package and post the source and binaries on the web site as a formal release • Announce the release via the email lists and web site *There is no set time limit on the development cycle. A complete cycle could happen in a matter of days or months 48 © Novell Inc. All rights reserved Development Process • Download the source code – – – – • Collaborate – – – 49 Most Open Source projects allow anonymous access to the source code repository CVS and Subversion are two of the more popular revision control systems Variety of client tools (WinCVS, TortoiseCVS, SVN TortoiseSVN) HTTP view access is also available for most projects Post messages to various email lists depending on the topic (Bugs, Development, Support, etc.) Monitor projects activity and progress by subscribing to a project’s mailing lists Easiest way to contribute to an Open Source project © Novell Inc. All rights reserved Development Process • Report bugs or suggest enhancements – – – – – • Submit a patch – – – 50 Submit a bug report through the tracking system or via project’s mailing list Bugzilla is the most popular bug tracking system for OSS Include as much detailed information as possible If possible include test cases that demonstrate the problem Requests for enhancements can also be submitted in the same manner Create a patch file according to the instructions outlined by the project (usually a simple DIFF listing) Submit the patch file through the project’s email list A patch can be submitted against source code or documentation © Novell Inc. All rights reserved Open Source Community Etiquette • Open source is a UNIX/Linux world – • Stay positive – • These are volunteers, they aren’t paid enough to be abused by anybody; “You’ll catch more flies with honey than vinegar” Contribute – 51 If an implementation doesn’t make sense in the nonUNIX/Linux world, suggest a way to enhance it rather than criticize it The community is interested in what you can contribute no matter what that contribution may be © Novell Inc. All rights reserved Secret to Success… • 52 How to succeed in the Open source community – Relationships – Relationships – Relationships • The key to success in the Open source community is forming good relationships through consistent participation, contributions and acknowledgements • Relationships are built on trust. Trust in openness and honesty, trust in the abilities, trust in reaching the common goal © Novell Inc. All rights reserved Resources 53 • http://developer.novell.com • http://forge.novell.com • http://www.oreillynet.com/pub/a/oreilly/opensource/ne ws/myths_1199.html • http://www.linux-foundation.org • http://www.catb.org/~esr/writings/cathedral-bazaar • http://www.nswscl.org.au/journal/51/Mark_H_Webbink. html © Novell Inc. All rights reserved References • The Art of Unix Programming by Eric S. Raymond – • The Cathedral and the Bazaar – • http://www.opensource.org/docs/definition.php Open Source Licenses – 54 http://www.catb.org/~esr/writings/cathedral-bazaar/ The Open Source Definition – • http://www.faqs.org/docs/artu http://www.opensource.org/licenses/ © Novell Inc. All rights reserved Questions Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.