Friday, December 08, 2006

XP - The New Way to Develop Software

Introduction –
Object-oriented programming using the Java language has become immensely popular. It has revolutionized software development to some degree, but recent studies show that half of software development projects are late, and one-third are over budget. The problem isn't the technology; it's the way software is developed. So-called "lightweight" or "agile" approaches, coupled with the power and flexibility of object-oriented languages like the Java language, offer an intriguing solution. The most popular agile approach is called Extreme Programming, or XP. Using XP on OOPS language projects can increase the chances of success dramatically.
Extreme Programming (XP) is a deliberate and disciplined approach to software development. We found XP to be successful because it stresses on customer satisfaction and the methodology is designed to deliver software as per customer needs and when it is required. XP empowers our developers to confidently respond to changing customer requirements, even late in the life cycle. XP prescribes a core set of values and practices that allow software developers to do what they do best: write code. XP eliminates the unnecessary artifacts of most heavyweight processes that distract from that goal by slowing down and draining the development staff (for example, Gantt charts, status reports, and multi-volume requirements documents).
This methodology also emphasizes team work. Managers, customers, and developers are all part of a team dedicated to delivering quality software. XP implements a simple, yet effective way to enable groupware style development.
XP improves a software project in four essential ways; communication, simplicity,
feedback, and courage. Our XP programmers communicate with our customers – proxy or real and fellow programmers on a continuous basis. Our designs are kept simple and clean and we get feedback by testing the software starting on day one. Deliveries of the system to the customers starts early and changes are implemented without the traditional problems of change management as in Classical SDLC. With this foundation our XP programmers are able to courageously respond to changing requirements and technology.

The anxiety about what XP can do to a development process is a typical example of resistance to change from the structured Classical SDLC with its phased development which last for long periods of time and defined stages to a programming concept where the entire SDLC can actually be developed in a single day. As an analogy, its like the early days of Java development. There were programmers who understood object-oriented programming and took advantage of some of the facilities especially inheritance; however, there were many more programmers who ported their C code to the Java language and then announced that they were developing as per OOPS concepts, which led to serious repercussions. Technically, these developers were doing object-oriented programming, but the approach - building one huge object that contained all the code that used to be embedded in their procedural programs - resulted in a serious hit to performance.

The 12 practices of XP –

Extreme Programming, or XP, is constructed on 12 basic practices that given below and for the most part, these basic practices are rarely demanding or difficult to use and follow.
1. The Planning Process - allows the customer to define the business value of desired features, using cost estimates provided by the programmers to choose what should be done and what should be deferred. XP planning addresses two key questions in software development: predicting what will be accomplished by the due date, and determining what to do next. The emphasis is on steering the project rather than on exact prediction of what will be needed and how long it will take. There are two key planning steps in XP, addressing these two questions:
Release Planning is a practice where the Customer presents the desired features to the programmers, and the programmers estimate their difficulty. With the costs estimates in hand, and with knowledge of the importance of the features, the Customer lays out a plan for the project. Initial release plans are necessarily imprecise: neither the priorities nor the estimates are truly solid, and until the team begins to work, no one can effectively predict just how fast they will go. Even the first release plan is accurate enough for decision making, however, and XP teams revise the release plan regularly.
Iteration Planning is the practice whereby the team is given direction every couple of weeks. XP teams build software in two-week "iterations", delivering running useful software at the end of each iteration. During Iteration Planning, the Customer presents the features desired for the next two weeks. The programmers break them down into tasks, and estimate their cost - at a finer level of detail than in Release Planning. Based on the amount of work accomplished in the previous iteration, the team signs up for what will be undertaken in the current iteration.
These planning steps are very simple, yet they provide very good information and excellent steering control in the hands of the Customer. Every couple of weeks, the amount of progress is entirely visible. There is no "ninety percent done" in XP: an application was completed, or it was not. This focus on visibility results in a nice little paradox: on the one hand, with so much visibility, the Customer is in a position to cancel the project if progress is not sufficient. On the other hand, progress is so visible, and the ability to decide what will be done next is so complete, that XP projects tend to deliver more of what is needed, with less pressure and stress.
2. Small Releases - means the developers put a simple system into production early and update it frequently on a short cycle. Releases should be as small as possible while still delivering enough business value to make them worthwhile. XP suggests that Releases should be as soon as it makes sense to do so. This provides value to the customer as early as possible. Small releases also will provide concrete feedback to developers on what meets customer needs and what doesn't. The team then can include these lessons in its planning for the next release.
3. Metaphor - means the team uses a common "system of names" and a common system description in development and communication. Extreme Programming teams develop a common vision of how the program works, which is called the "metaphor". At its best, the metaphor is a simple evocative description of how the program works. XP teams use a common system of names to be sure that everyone understands how the system works and where to look to find the functionality one is looking for, or to find the right place to put the functionality one is about to add. The system metaphor in XP is analogous to what most methodologies call architecture. The metaphor gives the team a consistent picture they can use to describe the way the existing system works, where new parts fit, and what form they should take.

4. Simple Design - The program should be the Simplest Design that meets the current requirements - without much thought about future versions. (That doesn't mean the program shouldn't scale or be inflexible.) The classical SDLC heavyweight approach say that even the very trivial design tasks has to be accomplished up front. XP says design should not be done all at once, up front, under a delusion that things won't change. XP considers design so important that it should be a constant affair. XP methodology always tries to use the simplest design that could possibly work at any point, changing it as the development proceeds to reflect emerging reality. The simplest design should follow the basic premises given below –
a. Runs all the tests
b. Contains no duplicate code
c. States the programmers' intent for all code clearly
d. Contains the fewest possible classes and methods

5. Acceptance Test Plans – First the Test Plans are written, then the applications are tested and validated to whether the software passes the test. Extreme Programming is obsessed with feedback, and in software development, good feedback requires good testing. XP teams practice "test-first development", working in very short cycles of adding a test, then making it work. Almost effortlessly, teams produce code with nearly 100 percent test coverage, which is a great step forward in most shops. These "programmer tests", or "unit tests" are all collected together, and every time any programmer releases any code to the repository (and pairs typically release twice a day or more), every single one of the programmer tests must run correctly. This means that programmers get immediate feedback on how they're doing. Additionally, these tests provide invaluable support as the software design is improved. The point is simple. Writing tests first ensures
· The most complete set of tests possible
· The simplest code that could possibly work
· A clear vision of the intent of the code
6. Refactoring - With Refactoring, the team improves the design of the system throughout the entire development. The refactoring process focuses on removal of duplication which is a sure sign of poor design. The result is that XP teams start with a good, simple design, and always ens up with a good, simple design for the software. This lets them sustain their development speed, and in fact generally increase speed as the project goes forward. Refactoring has to be strongly supported by comprehensive testing to be sure that as the design evolves, nothing is broken.

7. Pair Programming - In Pair Programming, all production code is written by two programmers working together at one machine. There are studies to demonstrate that this method produces better software at the same or lower costs than using lone programmers. This practice ensures that all production code is reviewed by at least one other programmer, and results in better design, better testing, and better code. Pairing, in addition to providing better code and tests, also serves to communicate knowledge throughout the team. As pairs switch, everyone gets the benefits of everyone's specialized knowledge. Programmers learn, their skills improve, they become move valuable to the team and to the company.

8. Collective Ownership - Each piece of code is subject to Collective Ownership, so any programmer can alter any piece of code with proper use of a tool to monitor the changes. All the contributors to an XP project sit together, members of one team. This team must include a business representative - the "Customer" - who provides the requirements, sets the priorities, and steers the project. It's best if the Customer or one of her aides is a real end user who knows the domain and what is needed. The team will of course have programmers. The team will include testers, who help the Customer define the customer acceptance tests. Analysts may serve as helpers to the Customer, helping to define the requirements. There is commonly a coach, who helps the team keep on track, and facilitates the process. There may be a manager, providing resources, handling external communication, coordinating activities. None of these roles is necessarily the exclusive property of just one individual: Everyone on an XP team contributes in any way that they can. The best teams have no specialists, only general contributors with special skills. Any person on the team should have the authority to make changes to the code to improve it. Everybody owns all the code, meaning everybody is responsible for it. This technique allows people to make necessary changes to a piece of code without going through the bottleneck of an individual code owner. The fact that everybody is responsible negates the chaos that ensues from no code ownership.

9. Continuous Integration - With Continuous Integration, several times a day, progress is rapid and many integration problems are eliminated. Infrequent integration leads to serious problems on a software project. First of all, although integration is critical to shipping good working code, the team is not practiced at it, and often it is delegated to people who are not familiar with the whole system. Second, infrequently integrated code is often full of bugs. Problems creep in at integration time that are not detected by any of the testing that takes place on an unintegrated system. Third, weak integration process leads to long code freezes. Code freezes mean that you have long time periods when the programmers could be working on important shippable features, but that those features must be held back.

10. 40 Hr week - Tired programmers make more mistakes, so generally the team is limited to work for 40 hrs every week.

11. On-site Customer – Its is better to have an On-site Customer available with the authority to determine requirements, set priorities, and answer questions or in his absence a suitable authority who can check the progress and see if it confirms to the requirement.

12. Coding Standards – It is essential to establish a Coding Standard, so programmers can meet the requirements of the other practices as this type of development requires continuous interaction between the programmers. Having a coding standard does two things:
· It keeps the team from being distracted by stupid arguments about things that don't matter as much as going at maximum speed.
· It supports the other practices.
Without coding standards, it is harder to refactor code, harder to switch pairs as often as one should, and harder to go fast. The goal should be that no one on the team can recognize who wrote which piece of code. The goal isn't to have an exhaustive list of rules, but to provide guidelines that will make sure the code communicates clearly. The coding standard should begin simply, then evolve over time based on team experience.

My Experience

Some of the above are essential to follow and followed as such at our development factory, some are followed on a case to case basis depending on the development patterns. Some of the practices not always used include –

No big up-front design – This is not always possible especially in large development with a vast database structure and most of our clients want to see a design before commencing the development – however, the entire development can be broken down into a series of iterative steps.

Metaphor and stories for coding – Not always required – its importance increases with the number of interdependent coding teams. Generally we ensure that Metaphors are utilized during any development process consisting of more than 4 paired teams over a time period exceeding 40 working days.

The 40-48-hour work week – We believe that human beings, especially as intelligent as our coders are talented enough to understand when they are tired and they need a break, however over a long period, we generally ensure and follow the 40-44 hr/week regime to ensure that all the programmers are fresh to undertake the necessary work-load.

· An on-site customer - Very often, we use a separate Project Manager, who is generally a very senior person, as the proxy for an on-site customer with one additional modification - The Project Manager does not drive or monitor the project development in any way. As part of presenting each desired feature, the XP Customer or his proxy defines one or more automated acceptance tests to show that the feature is working. The team builds these tests and uses them to prove to themselves, and to our customer, that the feature is implemented correctly. All our tests are automated – this is important because in the press of time, manual tests are skipped. Our XP teams ensure that once the test runs, the team keeps it running correctly thereafter by our rigorous Regression Testing Techniques. This means that the system only improves, always notching forward, never backsliding.

Frequent changes – If customers want to add features or change requirements, they are generally are allowed to depending on the complexity of change. The Management team just re-prioritzes features, but we believe that suggested changes should held until the next iteration.

Pair programming - falls into the "sometimes" bucket. One can easily visualize the advantages of pair programming during the design and algorithm-construction phase, but the efficacy during the coding and production phase is subject to actual environment. Also, very often the Project Manager alters the pair-programming rule in another way by assigning the pairs himself. He assign two programmers to work on a set task for one to three weeks. He meets with them and gives them the problem and the architected solution and sets them loose.

Advantages of XP -

Simple and elegant code – Software, which is engineered to be simple and elegant is more valuable than software that is complex and hard to maintain and XP methodology generally throws up simple and easily comprehensible code.

ROI – Our experience in Software Development has shown that a typical project will spend about twenty times as much on people than on hardware. That means a project spending 2 million dollars on programmers per year will spend about 100 thousand dollars on computer equipment each year. Let's say that we find a way to save 20% of the hardware costs by some very clever programming tricks. It will make the source code harder to understand and maintain, but we are saving 20% or 20 thousand dollars per year, which is a big saving. Now what if instead we wrote our programs such that they were easy to understand and extend. We could expect to save no less than 10% of our people costs. That would come to 200 thousand dollars, a much bigger savings. This is certainly something our customers appreciate.
Bugs - Another important issue to customers are bugs. XP emphasizes not just testing, but testing well. Tests are automated and provide a safety net for programmers and customers alike. Tests are created before the code is written, while the code is written, and after the code is written. As bugs are found new tests are added. Our strong Regression Testing methodologies sometimes to about 30% of code ensures bugs rarely get through twice.
Changing Requirements - XP enables us to embrace change. Too often we have found a customer will see a real opportunity for making a system useful with some changes after it has been delivered. XP short cuts this by getting customer feed back early while there is still time to change functionality or improve user acceptance. XP is especially useful when customers may not have a firm idea of what the system should do.

Project Risk - XP was also set up to address the problems of project risk. If customers need a new system by a specific date the risk is high. If that system is a new challenge for any software group the risk is even greater. If that system is a new challenge to the entire software industry the risk is greater even still. The XP practices are set up to mitigate the risk and increase the likelihood of success due to tight and controlled iterations, continuous feedback and repeated tests.
Smaller teams - We use XP generally for small groups of programmers - between 2 and 12, though we have occasionally used this methodology for larger projects of 30 or more people with success. We have found that on projects with dynamic requirements or high risk a small team of XP programmers will be more effective than a large team.

Continuous Interaction - XP requires an extended development team. The XP team includes not only the developers, but the managers and customers as well, all working together elbow to elbow. Asking questions, negotiating scope and schedules, and creating functional tests require more than just the developers be involved in producing the software.
Testability – Our testing methodology is geared to create automated unit and functional tests. Sometimes, we change your system design to be easier to test, but at the end of the day, every functionality is tested - where there is a will there is a way to test.
Productivity - The last thing on the list is productivity. XP projects unanimously report greater programmer productivity when compared to other projects within the same corporate environment. But this was never a goal of the XP methodology. The real goal has always been to deliver the software that is needed when it is needed.

No comments:

The GOATs of Hindi film music - an analysis of the greatest music directors in the history of Bollywood cinema

Over the holiday break in December 2020, I did a fair bit of driving around the Australian east coast. The drive from Sydney up to central Q...