Techdoer Times

Previously, I compared the performance of today’s popular operating systems with respect to multi-core processors.  In this final part to Challenges in Multi-Core Era, I’ll talk about the multi-core capabilities found in today’s programming languages and development tools.

The Programming Languages

When language architects were designing the foundations of the most popular programming languages, multi-core microprocessors were hidden in laboratories. Only high performance servers had access to multiprocessing systems. Just a few specialized workstations had more than one CPU installed. Therefore, C# and Java offered support for concurrency and multi-threading intended to offer more responsive applications. However, language architects didn’t design this support to optimize applications for future multi-core CPUs. Hence, nowadays, it is really difficult to optimize existing code to take advantage of multi-core CPUs using frameworks prepared for serial code.

In order to take full advantage of multi-core, the applications need a task-based design and a task-based programming. There is no silver bullet. So far, there is no way to optimize an application recompiling it without changes. Some developers expected this to happen. There is a great need for new designs, new programming techniques and new tools. The software development industry needs a new great paradigm shift.

Besides, developers need a framework capable of handling tasks. There are new programming languages, or new versions of older concepts, like functional programming. Functional programming makes it easier to code task-based designs and to split the work to be done in multiple independent tasks that could be run in parallel on multi-core CPUs.

There are many new programming languages with a great focus on functional programming, prepared to take full advantage of multi-core and to offer a great scalability. Just to mention a few:

• Scala
• Haskell. Yes, the one that has more than twenty years. Pure functional programming languages are back and they can be the future for parallel programming.
• Microsoft Axum (formerly Maestro).
• Microsoft F#.

However, do developers want to begin learning new programming languages? Most developers want to leverage their existing knowledge to move onto multi-core programming.

C++ and Fortran programmers had early access to parallel programming. Nowadays, these are the only programming languages that can take advantage of the full power offered by modern microprocessors. C++ is closer to the hardware. Hence, it allows many optimizations that aren’t available in any other programming language – apart from C and assembler. You can access all the vectorization capabilities from C++ and Fortran.

OpenMP has been offering a high quality multi-platform and open source shared-memory parallel programming API to C/C++ and Fortran for many years now. Besides, Intel Threading Building Blocks, also known as TBB, allows developers to express parallelism in C++ applications to take advantage of multi-core.

Message Passing Interface (MPI), is a language-independent communications protocol used to program parallel computers. You can use MPI to take advantage of multi-core on many programming languages. However, its main focus is to help develop applications to run on clusters and high performance computers.

Intel offers compilers and libraries optimized to take advantage of multi-core. However, you still require coding your applications considering new parallel designs. The usage of vectorization in their math libraries is a great plus point. There is an outstanding opportunity for new libraries and components optimized for multi-core and vectorization. Parallelism brings new opportunities to the software development industry.

There are new companies taking advantage of the need for multi-core optimizations, like Cilk Arts, offering its Cilk++ compiler. It is based on GCC and includes a modified compiler and debugger to simplify multi-core programming for Linux and Windows platforms.

Mac OS X’s Xcode development environment offers access to Grand Central Dispatch and OpenCL. OpenCL allows C programs to run in the GPU instead of loading the main CPU. The interest of developers in Xcode has really grown since multi-core and OpenCL.

C# and Java are evolving in order to offer developers new ways of expressing parallelism in their code. Indeed, they are changing many aspects that were designed for another world, the old single-core machines.  Some of these changes include new garbage collectors, new frameworks and features, new functional approaches and task-based programming capabilities.

Java 7 will offer the new fork-join framework, really optimized for multi-core.

C# 4.0 (Visual Studio 2010) will add task-based programming capabilities and parallelized LINQ (PLINQ). Besides, it will allow the possibility to manage the desired degrees of parallelism.

Furthermore, there are new DSLs (Domain Specific Languages) to add parallel programming capabilities to existing high-level languages. For example, GParallelizer adds nice parallel programming capabilities to Groovy.

Most modern programming languages are evolving or adding inter-operatibility capabilities with other languages to favor multi-core programming.

However, don’t forget about vectorization. Mono, a free and open source .Net compiler, offers access to SSE3 and SSE4 for C# developers.

A few years ago, concurrency was about threads. Now, experts are talking about tasks and fibers. Why? Because in order to develop an application using a task based approach, threads are too heavy. Tasks and fibers are lightweight concurrency elements, much lighter than threads. They allow developers to implement complex task-based designs without the complexities of threads.

Ruby 1.9 added fibers and it simplify the creation of pipelines. Pipelines take great advantage of Hyper-Threading combined with multi-core.

If you want to go parallel, follow James Reinders’ eight key rules for multi-core programming. You can apply them to any combination of programming language, framework, compiler and virtual machine.

Java 7 and .Net 4 will not offer framework support for vectorization (SIMD instructions). SIMD has been there since Pentium MMX microprocessor.  This decision doesn’t make sense especially considering that there is a huge market for smart professionals in the new parallelism age.

The New Tools

A well-known proverb says “A good workman is known by his tools”.

A parallelized application requires new debugging and testing techniques. You need to catch potential bugs introduced by concurrency.

Intel has been offering tools for High Performance Computing and parallelism for many years now. A few weeks ago, Intel launched one of the most complete parallel toolkits for C/C++ developers, Intel Parallel Studio. Among many other features, it helps developers to compile applications tuned for multi-core CPUs and to find concurrency specific bugs and bottlenecks. You should expect to see more tools like this coming on the next quarters.

Visual Studio 2010 will add enhanced multi-monitor support capabilities. You’ll need more than one monitor in order to debug applications running with a task-based approach. It will also add task-based debugging capabilities. However, Visual Studio 2010 has recently entered Beta 1. Therefore, if you want to develop an application using a task-based approach using C# 3.0, you still have to work with threads. Visual Studio 2008 offers nice multithreading debugging capabilities.

Most IDEs are changing to offer new task-based programming, debugging and testing capabilities. You have to test parallelized applications on multi-core CPUs. Many bugs aren’t going to appear when running them on single-core CPUs.

There are many free tools to help you in the multi-core jungle. You can monitor your applications and test their concurrency efficiency using Process Explorer and Intel Concurrency Checker.  If you use these tools to check commercial software, you’ll be able to see the need for new multi-core aware developers. Besides, you’ll see a lot of opportunities in the multi-core age.

By the way, multi-core programming has a high quality weekly talk show lead by Intel experts, Parallel Programming Talk.

Summary

While the old free lunch is over, the industry is reshaping itself to take advantage of the new microprocessors architectures of today and tomorrow.

Hardware will continue to advance and offer more parallel processing capabilities, even though the software industry is moving more slowly than expected.

Bottom line is that multi-core seems to be a really sustainable competitive advantage that requires a great paradigm shift from developers throughout the software lifecycle. There is light at the end of the tunnel. Are you ready to reach it?

About the author: Gaston Hillar has more than 15 years of experience in IT consulting, IT product development, IT management, embedded systems and computer electronics. He is actively researching about parallel programming, multiprocessor and multicore since 1997. He is the author of more than 40 books about computer science and electronics.


Gaston is currently focused on tackling the multicore revolution, researching about new technologies and working as an independent IT consultant, and a freelance author. He contributes with Dr Dobb’s Parallel Programming Portal, http://www.go-parallel and is a guest blogger at Intel Software Network http://software.intel.com.


Gaston holds a Bachelor degree in Computer Science and an MBA.You can find him in http://csharpmulticore.blogspot.com and http://software.intel.com/en-us/profile/417051/

Previously I talked about the evolution of microprocessors and specialized hardware since the wide-spread adoption of multi-core began a few years ago.  In this second part to Challenges in Multi-Core Era, I’ll compare the multi-core capabilities across today’s popular operating systems.

The Operating Systems

No matter the version, Mac OS always had a great advantage over any other desktop operating system. It knows exactly the underlying hardware because it is designed for running on Mac’s certified hardware only. You can run it on different hardware, at your own risk.  The same company develops the computers and the operating system. Leaving its great innovation aside, this is its great secret. For this reason, it can be tuned to take full advantage of specific hardware. For example, Mac OS X’s latest versions running over Intel microprocessors take advantage of vectorization, they use SSE (Streaming SIMD Extensions) and SSE2. In fact, Apple has been promoting vectorization and SIMD (Single Instruction Multiple Data) instructions in its Developer Connection Website.

However, Mac OS X Snow Leopard is giving another great step, offering Grand Central Dispatch.  Nonetheless, there is a big problem. There are too few developers working with specific Mac developer tools. Mac is also going to the 64-bits arena.

FreeBSD is one of the free and open source operating systems that always offered great features when working with multiprocessor systems. FreeBSD 7 works great with multi-core CPUs as well. Therefore, many high-performance servers around the world trust in FreeBSD’s scheduler.

The key is the operating system scheduler.  It is responsible for distributing the physical and logical processing cores, and assigning processing time to each concurrent thread.  It performs a really complex task.  For example, an operating system running over an Intel Core i7 CPU has eight logical processing cores (it can run eight hardware threads concurrently), but four physical cores. It has to distribute dozens of threads on time-slices available from eight logical cores. Thus, the scheduler efficiency impacts on the application’s performance. An inefficient scheduler could ruin a highly parallelized application’s performance.

Linux works great with multi-core CPUs, FreeBSD works better , but Linux does a great job. However, many desktop applications running on Linux GUIs aren’t optimized to take full advantage of multi-core. Nevertheless, Linux running as a Web server in a classic LAMP configuration is really fined tuned for multi-core.  Free Linux versions have a great advantage on the multi-core world as they don’t limit the number of cores using expensive licenses. Therefore, you can upgrade your desktop or server without needing to worry about the number of cores supported by your operating system license.

Both FreeBSD and Linux have a great advantage over Mac and Windows, most new deployments using these operating systems are using the 64-bits versions. Applications running in 64-bits offer more scalability than their 32-bits counterparts.  Parallel algorithms require more memory than serial ones. Most operating systems running in 32-bits can only address 4 GiB. There are some techniques to work with more memory even in 32-bits. However, they reduce the memory access performance.

4 GiB could seem a lot of memory. Nevertheless, as the memory map is a bit complex, you need some spaces for other purposes, like writing data to the video memory and other buffers. Hence, the operating system cannot access the whole 4 GiB of main memory. Furthermore, some operating systems limit the maximum memory addressable by an application in 32-bits mode (2 GiB in Windows standard configurations). Again, there are some techniques to work with more memory for the applications.

Whilst working with large images, videos and databases, these limits could be a great problem to add scalability. More cores mean more processes, more threads, more tasks or more fibers. A 2 GiB memory limit for a single process could mean a great I/O bottleneck.

Hence, working with 64-bits is a great step ahead. You have the same driver problems in both 32-bits and 64-bits Linux. Thus, you can install 64-bits Linux without worrying about additional problems. Working with 64-bits, you can scale as the number of cores increase without worrying about memory problems. Of course, you do have to worry about available physical memory. However, that’s another problem.

FreeBSD and Linux schedulers already offer an excellent support for NUMA. However, you may see nice improvements in future kernel versions. The idea is simple, the more the number of cores, the more optimizations required for an efficient scheduler.

Now, let’s move to Windows wonderland. Windows has a great disadvantage; each version offers different capabilities related to multi-core. You should check the maximum number of cores and the maximum memory supported by each Windows version. This doesn’t mean that Windows newest versions aren’t capable of handling multi-core CPUs. The problem is that they have different licenses and prices.

If you want scalability, you shouldn’t consider 32-bits versions anymore. Windows 2008 Server R2 and Windows 7 will support up to 256 logical processor cores. Nowadays, 256 logical cores seems a huge number. However, an Intel Core i7 is a quad-core CPU, but it offers eight logical cores (2 logical cores per physical core, 2 x 4 = 8). New CPUs with 8 physical cores are around the corner. Besides, they will offer 16 logical cores (2 logical cores per physical core, 2 x 8 = 16). Hence, you will see 16 graphs in your CPU activity monitor. An operating system offering support up to 256 logical cores really makes sense for the forthcoming years.

Despite being criticized everywhere, Windows Vista offered nice scheduler optimizations for multi-core. Windows 2008 Server R2 and Windows 7 will offer support for NUMA in their 64-bits versions. However, you must use the new functions to take full advantage of NUMA capabilities.

No matter the operating system, the applications must be prepared to take advantage of multi-core CPUs.
The operating systems are adding multi-core optimizations. However, except from Mac OS X, most applications running on the operating system take advantage of neither multi-core nor vectorization. I really cannot understand this.

I do believe it is time to learn from Mac OS X. For example, Windows 7 should offer a special version, let’s call it 7 Duo. 7 Duo could require at least a dual-core CPU with SSE2. So, you wouldn’t be able to run 7 Duo on older machines. If you have newer hardware, you’d buy and install 7 Duo. The operating system would load much faser, taking full advantage of your modern hardware. Your favorite Web browser should take advantage of multi-core and vectorization when parsing HTML or XML. Check this white paper, Parallelizing the Web Browser.

The great problem with PCs (x86 family) is the backward compatibility. It’s been an incredible advantage over the last decades. However, it is time to take advantage of modern hardware establishing baselines.
The same happens with Linux. I’d love to install an Ubuntu Duo on my notebook.

The operating systems are really crucial to tackle the multi-core revolution. Their schedulers are very important to transform multi-core power into application performance. Vectorization and SIMD is also important and most applications are not using it. It seems logical to develop new operating system versions designed to take full advantage of really modern hardware. They would add real value to your notebooks, desktop, workstations and servers.

Click here for part 3 where I compare the new capabilities of programming languages and development tools with respect to multi-core processors.

About the author: Gaston Hillar has more than 15 years of experience in IT consulting, IT product development, IT management, embedded systems and computer electronics. He is actively researching about parallel programming, multiprocessor and multicore since 1997. He is the author of more than 40 books about computer science and electronics.


Gaston is currently focused on tackling the multicore revolution, researching about new technologies and working as an independent IT consultant, and a freelance author. He contributes with Dr Dobb’s Parallel Programming Portal, http://www.go-parallel and is a guest blogger at Intel Software Network http://software.intel.com.


Gaston holds a Bachelor degree in Computer Science and an MBA.You can find him in http://csharpmulticore.blogspot.com and http://software.intel.com/en-us/profile/417051/

A few years ago, in 2005, Herb Sutter published an article in Dr. Dobb’s Journal , “The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software”. He talked about the need to start developing software considering concurrency to fully exploit continuing exponential microprocessors throughput gains.

Here we are in year 2009 – more than four years after Sutter’s article publication. What’s going on? How are we doing? How did the industry evolve to tackle the multi-core revolution?

In this three part series, we’ll answer these questions by exploring the recent multi-core inspired evolution of components throughout the application stack, including microprocessors, operating systems and development platforms.

The New Microprocessors

Microprocessor manufacturers are adding processing cores. Most machines today have at least a dual-core CPU. However, quad-core CPUs are quite popular on servers and advanced workstations. More cores are round the corner.

There is a new free lunch. If you have an application designed to take advantage of multi-core and multiprocessor systems, you will be able to scale as the number of cores increase.

Some people say multi-core wasn’t useful. You can take a look as this simple video. It runs four applications (processes) at the same time on a quad-core CPU. Each application runs in a different physical processing core, as shown in the CPU usage history real-time graph (it uses one independent graph per processing core). Hence, it takes nearly the same time to run four applications than to run just one. Running just one application takes 6 seconds. Running four applications takes 7 seconds. What you see is what you get. There are no tricks. Multi-core offers more processing power. It is really easy to test this. However, most of the software wasn’t developed to take advantage of these parallel architectures in single applications.

There is another simple video showing one application running on a quad-core CPU. The first time, it runs using a classic, old-fashioned serial programming model. Hence, it just uses one of the four cores available, as shown in the CPU usage history real-time graph. Then, the same application runs in a parallelized version, taking less time to do the same job.

In recent years, parallel hardware became the mainstream standard in most developed countries. The great problem is that the speed of hardware evolution went much faster than the speed of software evolution, resulting in a large gap between the two. The microprocessors added new features that software developers didn’t exploit. Why did this happen? Because it was very complex to accomplish it. By the way, it’s still a complex task. I’ll get back to this later.

However, the most widespread model for multiprocessor support, SMP (Symmetric Multi-Processor) leaves the pole position to NUMA (Non-Uniform Memory Access). On the one hand, with SMP, the processor bus becomes a limitation to future scalability because each processor has equal access to memory and I/O. On the other hand, with NUMA, each processor gains access to the memory it is close to faster than to the memory that is farther away. NUMA offers better scalability when the number of processors is more than four.

With NUMA, computers have more than one system bus. A certain set of processors uses each available system bus. Hence, each set of processors can access its own memory and its own I/O channels. They are still capable of accessing the memory owned by the other sets of processors, with appropriate coordination schemes. However, it is obviously more expensive to access the memory owned by the other sets of processors (foreign NUMA nodes) than to work with the memory accessed by the local system bus (the NUMA node own memory).

Therefore, NUMA hardware requires different kinds of optimizations. The applications have to be aware of NUMA hardware and its configurations. Hence, they can run concurrent tasks and threads that have to access similar memory positions in the same NUMA node. The applications must avoid expensive memory accesses and they have to favor concurrency taking into account the memory needs.

A new free lunch offers manycore scalability. Expect more cores coming in the next months and years. Learn about the new microprocessors. Be aware of NUMA and optimize your applications for these new powerful architectures.

The New Specialized Hardware

On the one hand, we have a lot of software that is not taking full advantage of the available hardware power. On the other hand, there are many manufacturers developing additional hardware to offload processing from the main CPU. Does this make sense?

This means that you are wasting watts all the time because you’re using obsolete software. In order to solve this problem, you have to add additional, expensive hardware to free CPU cycles. But, you aren’t using entire cores.

TCP/IP Offload Engine (TOE) uses a more powerful NIC (Network Interface Card) or HBA (Host Bus Adapter) microprocessor to process TCP/IP over Ethernet in dedicated hardware. This technique eliminates the need to process TCP/IP via software running over the operating system and consuming cycles from the main CPU. It sounds really attractive, especially when working with 10 Gigabit ethernet and iSCSI.

CPUs are adding additional cores. So far, modern software is not taking full advantage of these additional cores. However, you still need new specialized hardware to handle the network I/O… Most drivers don’t even take advantage of old parallel processing capabilities based on SIMD (Single Instruction Multiple Data) offered since Pentium MMX arrival. TCP/IP offload Engine is a great idea. However, if I own a quad-core CPU with outstanding vectorization capabilities, SSE4.2 and previous versions, I’d love my TCP/IP stack to take advantage of it.

Vectorization based on SIMD allows a single CPU instruction to process multiple complex data at the same time. Thus, using them, it speeds up the execution time of complex algorithms many times. For example, an encryption algorithm requiring thousands of CPU cycles could perform the same results requiring less than a quarter of these CPU cycles using vectorization instructions.

Something pretty similar happens with games. Games are always asking for new GPUs. However, most games take advantage of neither multi-core nor vectorization capabilities offered by modern CPUs. I don’t want to buy new hardware because of software inefficiencies. Do you?

Modern GPUs (Graphics Processing Units) are really very powerful and they offer an outstanding processing power. There are many standards to allow software developers to use these GPUs as a CPU, like CUDA and OpenCL. They allow the possibility to run general purpose code on the GPU to release the main CPU from this load. It sounds really attractive. However, again, most software does not take full advantage of multi-core. It seems rather difficult to see commercial and mainstream software considering the possibilities offered by these modern and quite expensive GPUs. Most modern notebooks don’t offer these GPUs. Therefore, I see many limitations to this technique.

Before considering these great but limited capabilities, it seems logical to exploit the main CPU’s full processing capabilities. Most modern notebooks offer dual-core CPUs.

Specialized hardware is very interesting indeed. However, it isn’t available in every modern computer. It seems logical to develop software that takes full advantage of all the power and instruction sets offered by modern multi-core CPUs before adding more specialized and expensive hardware.

In part two of Challenges in Multi-Core Era, I’ll compare the multi-core capabilities of the latest operating systems.

About the author: Gaston Hillar has more than 15 years of experience in IT consulting, IT product development, IT management, embedded systems and computer electronics. He is actively researching about parallel programming, multiprocessor and multicore since 1997. He is the author of more than 40 books about computer science and electronics.


Gaston is currently focused on tackling the multicore revolution, researching about new technologies and working as an independent IT consultant, and a freelance author. He contributes with Dr Dobb’s Parallel Programming Portal, http://www.go-parallel and is a guest blogger at Intel Software Network http://software.intel.com.


Gaston holds a Bachelor degree in Computer Science and an MBA.You can find him in http://csharpmulticore.blogspot.com and http://software.intel.com/en-us/profile/417051/

Lately I find myself caught in the middle of major sporting events.  I started the year in Argentina while the Dakar rally was moving its way through the vibrant streets of Buenos Aires.  A few months later I stumbled upon the 2009 champions league final and Giro d’Italia festivities in Rome.  Whether you’re a fan or not,  these sporting events provide a good opportunity to  learn from the psychology of the world class athletes and teams behind them.

For instance, an recent article in a pro cycling journal serves as a good reminder that sometimes moving forward requires letting go.  As software practitioners we’ve all probably experienced the frustration that comes with holding on to overly complex development environments, inefficient development processes, or seeking new job opportunities based strictly on prior experience alone.   In the latter case, a Java developer may unnecessarily limit a  job search to only those positions matching his or her current skillset,  not realizing the attractiveness offered by development opportunities involving newer technologies such as Ruby on Rails.  In recent years, the Rails community has made enormous strides towards making software development simple and fun again.

In this three part series, I’ll share my first impressions as I let go of prior knowledge investments in J2EE and .NET in favor of this exciting new ecosystem of software development I refer to as “Ruby Land”.  I’ll show how long standing best practices in software engineering  have been injected into the core culture of its practitioners, and where the lines between artist, entrepreneur, and programmer have blurred in favor of promoting the human side of software development, with an acute focus on continuous testing, productivity, and programmer happiness.

Let’s Rewind a Bit

It may be unfair to single out Java, but the start of this decade saw some consensus among the J2EE development community (2002, 2004, 2006) that the technology’s focus on enterprise computing  had rendered it increasingly complex, cumbersome and dissatisfying.  Am I the only technologist who ran the other way with the introduction of EJB application servers?

Java has since evolved by leveraging the strong foundation and support of its Virtual Machine with the introduction of dynamic languages such as JRuby and Groovy.

Nevertheless, by the middle of this decade, a window of opportunity had opened, permitting a stream of defections from development  communities whose technology had disenchanted its practitioners.

An interesting quote from Charles Connel says:

Junior programmers create simple solutions to simple problems. Senior programmers create complex solutions to complex problems. Great programmers find simple solutions to complex problems.

The one defining characteristic of the Ruby/Rails community seems to be the high concentration of great programmers – professionals who continuously seek out simple technologies to help solve their complex problems.  As Martin Fowler put it:

Ruby has a philosophy of an environment that gives a talented developer more leverage, rather than trying to protect a less talented developer from errors. An environment like Ruby thus gives our developers more ability to produce their true value.

Programmers At Work

Each generation of software development has always had its share of great programmers.   Early on we had the programmers at work – the early pioneers who blazed the trail we continue to walk on today.

When I first entered the professional world in the 90’s, I remember admiring names like Gosling, Bosworth, Bray, Wall, Ozzie, Booch, Lee, Linus, Raymond, Sessions, O’Reily, Box, and McConnell.

By the middle of this decade this list had grown to include people like  Joel, Brinn, Page, Fowler, and Graham.

The newest generation of Ruby/Rails programmers is not only limited to engineers and entrepreneurs but artists as well.  As newer languages climbed the ladder of abstraction in recent decades, the programming discipline began appeal to a wider more diversified audience of professionals. Today you find artists, engineers, and entrepreneurs all involved somehow in the task of building software systems or promoting and growing the business of software.  Names and monikers include DHH, Uncle Bob, Dr. Nic, and Bates.

These programmers have distinguished themselves by providing exceptional solutions, in the form of code, tutorials, and conferences, that keep Ruby/Rails anchored to its core values while helping grow its usefulness and adoption.

Check back soon as we continue this series Leap Forward with Rails.

The testing of software products or any software artifact, after the fact (i.e. inspection to find defects versus inspection to prevent defects), adds no value to the business providing the product or customer using the software.  In this regard we can say that testing is waste.

What!? Should I stop testing, then?

Well no, but let’s discuss from a couple of perspectives.

Am I cheating my customer?

The story goes like this:

• My customer wants a high-quality product
• Because we are human, we make mistakes and therefore no software is bug-free
• We test in order to catch problems before they reach the user.

So far so good, but now comes the difficult part. Determining how much to test is an inexact science.  I can infinitely test the software driving up project costs throughout. I could also let the client decide how much to test (I am agile after all), but beware! His decisions could result in tons of technical debt in the form of untested code. I could also carefully explain the needs, alternatives and tradeoffs in the various testing approaches, how much automation to incorporate; this would all a piece of cake. In fact, I have just convinced him to pay for the testing, and now you tell me that testing is a waste. Am I making my customer pay for waste?

We are professionals that take pride in our work. We strive not only to be effective but to also be efficient. Our aim is to perform only those activities that contribute to the final outcome. Thinking we are wasting customer money is disturbing. Just bear with me a little longer.

Not my fault, this is a new concept

I could just tell my customer “I’m sorry but I just realized that testing is a waste.”

Truthfully, the idea of testing as something to avoid has been with us for quite long time. In Software Process Improvement (SPI) the story of “testing is waste” goes as follows.

To build a piece of software, we first gather requirements, analyze them, design the solution, program and implement the design and finally deliver it. Now suppose that the user finds a defect after using the product. How much would cost?

Let me introduce cost of quality (CoQ) and its taxonomy. CoQ relates to the costs which are incurred due to a lack of quality and costs which are incurred in the achievement of quality. Cost due to lack of quality is caused by failure before (internal) and after (external) the delivery and the cost of fixing the defect. Cost of achieving quality includes appraisal (inspection) and prevention costs.

Some defects could be mapped to a problem that originates during the programming or design stages. Could we have detected closer to the moment the defect originated? How much cost would we have incurred in this case?

Barry Boehm published in ’81 a much cited study saying that a change in the requirement phase cost x1, in analysis x10, in design x100, and so on. In the context of Boehm (bureaucratic environments, projects with fixed scope, mostly US Government and contractors to them), most of the changes come from defects.  Then, taking quality upstream makes economic sense.

CoQ was used to justify Big Requirement Up Front and the whole BxUF, where x is also analysis, design, etc. If you do it right from the beginning, what could go wrong?  This is the infamous waterfall approach.

But you as well as everyone else know that the universe throws us curve balls all the time in the form of changing requirements. As a result, we have to iterate. Iteration means rework, and rework is bad, but what can we do?

What about automating the whole development process itself? You know, Computer Aided Software Engineering (CASE) and all that stuff. Old fashion and will surely have some problems, but remember the goal is to make it right from the start. This would reduce the CoQ.

You have to be politically careful in attacking CoQ; it is used as an economic justification for SPI initiatives including CMMI projects, but also to justify agile methods.

The exact factors influencing each phase in the cost of change curve are not collectively agreed to, but Alistair Cockburn argues that the curve is correct even in XP.  So the tool is not bad per se, but it could be used for justifying wildly different solutions.

The machine that changed the world

You are probably thinking “Moving testing upstream seems reasonable; we want to make every task as efficient as possible. That still doesn’t mean that the testing will add value!”

Sometime we find ourselves arguing just because we assign different meaning to the same words. Value, Waste … let’s agree what they mean to me, but first I will digress.

Japanese car manufacturers (starting with Toyota) use a set of principles and practices called Toyota Production System, currently know as Lean Production, radically changed the way that production is organized. The Lean principles are:

• Eliminate waste
• Amplify learning
• Decide as late as possible
• Deliver as fast as possible
• Empower the team
• Build integrity in
• See the whole

Just-In-Time and Stop-the-line (Jikoda) practices, which originate from Lean Production, were counter-intuitive when first proposed, but nowadays those and other Lean Production practices are widely accepted and considered to have had such a positive impact to be referenced in The Machine That Changed the World.

But Lean Production is about manufacturing. Software development is not manufacturing. Does Lean Production apply to Software Product Development? Not directly, you must take a look at The Toyota Product Development System to see how the principles are applied. And finally, there are difference between developing cars and software.

You may already know Lean Software Development and Mary and Tom Poppendieck’s work. They applied Lean principles to software development. This is a pretty new area, still in motion, but it already having a positive impact in the Agile community, triggering many interesting discussions.

Value Stream Mapping

In Lean, optimization must be global (“See the whole” principle).

One of the Lean tools for detecting opportunities for improvement maps the flow of a “piece” thru the value adding system (From concept to cash), taking note of the time spent by the piece in each step of the flow and whether any queue of pieces or work in process (WIP) exists. The flow’s end-to-end time is called lead time. Having the map, you can identify improvement applying mostly two principles: Eliminate waste and shorten the end-to-end time (Deliver as fast as possible). The nice thing is that shortening the lead time forces you to eliminate waste.

What is a piece in software development? You can think in a new functionality, from the moment that is conceived to the moment that starts to be used by a user in a production environment.

After identifying all kinds of waste, you can measure the efficiency of your process as the ratio between the time spent in value adding tasks and the lead time.

Eliminating waste is a never ending activity. Processes will never have 100% efficiency.

Waste …

Waste is everything that doesn’t add value as perceived by the user.

There are many kinds of waste. There are some common waste patterns: every queue and waiting step is waste, and also every rework or product not used/sold is waste. For instance in software there are 7 types of waste:

• Partially Done Work
• Extra Features
• Lost Knowledge
• Handoffs
• Task Switching
• Delays
• Defects

But even if defects can be classified as waste (they are both a queue and a signal that rework is needed), you can still argue that detecting them thru testing is not waste.

No, no. You must work in such a way that defect didn’t happen in the first place (Build quality in principle). And if you can’t do that, at least you must build incrementally so small that you detect the defect almost when it is happening, and fix it (and it cause) immediately. You know where I’m going: Test-driven development (TDD) and Acceptance Test-Driven Development (ATDD). The catch is that testing after the fact (inspection after the defect occurs) is waste, while testing to prevent the defect is not waste.

This is easier to understand in manufacturing, but what about software development?

There is consensus that TDD is not testing but a design that could be executed and validated automatically, so it is not waste. But ten years ago most of us thought that a good set of UML diagrams was the best representation of the design and that there was value in keeping it updated.  Who knows, in ten years we might know the “proper” way of doing design, and tests associated with TDD will be waste used only by dinosaurs.

… and Value

Software testing is a measurement of the qualitative attributes of the product. Once tested, you’ll have a better understanding of the product, but from the customer’s point of view, it is still the same product.

In some cases the customer may request and pay for a particular measurement (test). Is this a case where testing adds value? The answer is no. It is more likely a lack of trust which causes the customer to request and pay for this wasteful behavior.

After all, what good is in testing?

Some care must be taken. Saying that something is waste doesn’t mean that you can just get rid of it. After analyzing the value stream mapping of your organization, you might find some quick benefits, like removing duplicate tasks (i.e. testing by the development teams and by an external test team), but many times you can remove waste only after changes occur in other activities or tasks. Take, for example, the lack of trust situation. Removing tests is only possible if the customer trusts us, and this will take time. Other common examples: TDD or ATDD could decrease the amount of testing needed at the end, but you have to change how you manage requirement and how you design.

Every test is waste?

Brian Marick proposes a classification of test in four quadrants. In one dimension you have whether the tests “support the team” or “critique the product”. The other dimension is whether the tests are “technology-facing” or “business-facing”.

Most (or all?) tests that support the team are potentially automated and test-first-able. If you find yourself doing this kind of test after the fact or manually, it’s waste.

But what about tests that critique the product? While doing them (i.e. exploratory testing) you are learning the emergent properties of the product, you are finding ways to improve the product. It is just too easy to say that those are not waste but value adding activities. Don’t. Read Agile Testing, of Lisa Crispin and Janet Gregory. Make some experiments. Some of those tests could be done before having a product.

I will lose my job!

If testing is in fact waste, what does that mean for us poor testers! It can’t be true, or not always or not everybody will do the right things.

I’m trying to shake you a little, but is your current way of developing software (process has a bad name lately) bad?

I think that usually the answer is no. It is the best way … until now. You can always improve it.

Don’t give up hope, read Testers: The hidden resource by Lisa Crispin and Janet Gregory. There are many ways in which a person coming with a testing background could contribute to an agile team.

Conclusion

Quality is paramount in agile development; we need quality to have effective and efficient teams. But quality is not testing. Testing is commonly done by people with great commitment to quality, but is the least efficient moment to add quality to a product.

The team needs quality champions, but those people (testers like me) could feel out of place in an agile team. As testers, we need to leverage our skill, help the team find ways to include quality early on. As testers, we have to work hard for a team that doesn’t need after the fact testing, cannibalizing our experience, the same way that successful companies do with products.

We are living interesting times; we are in the middle of paradigm change (not to mention some financial problems). If you define your value as professional based in your ability to do a particular task, you are at risk. I try to identify how I contribute to software development at a higher level, and I keep learning. I hope that this is the answer!

What is your answer?

About the author: Juan Gabardini has more than 20 years of experience in IT management, IT product development, teaching and consulting in financial services, retail, telecommunications, medical equipment and manufacturing sectors.


Juan is currently focused on testing and coaching the Agile way, and building an Argentinian and Latam Agile community (Agile Open Tour and Ágiles 200x).


Juan is a member of IEEE, Agile Alliance and Scrum Alliance, and holds degrees in System Analysis and Electronic Engineering from the Buenos Aires University, where he also is teaching. You can find him (in Spanish) in http://softwareagil.blogspot.com and http://www.agiles.org

Effective Destabilization

Software practitioners familiar with the rules of extreme programming know that moving people around, or the process of continually appointing team members to work on different parts of the solution or to serve in different team roles, can effectively shuttle knowledge between team members helping to ensure a common level of understanding in both the problem and solution domains.

Moving people around serves as a force that effectively destabilizes the team and continually engages them  towards establishing a new comfort zone.  In the process, team members cross-train towards improving their skills, are deterred from complacency, become more aware of project status, while becoming more responsive to project risk.  In this series, Moving People Around, I’ll discuss the challenges and opportunities that resulted from my own experience with a similar destabilizing force as I moved from a colocated team setting in New York City, to a teleworking setting in Buenos Aires, Argentina.   I’ll discuss how our team overcame the resulting challenges and amplified the opportunities towards ensuring the successful delivery of our software projects.

Surging Ahead

Prior to my moving, our team was composed of 4 members, 3 superb programmers and 1 tech lead/hands-on project manager (me).   Our efforts were focused on delivering a .NET analytics platform which was in beta release.

We regularly incorporated agile software development practices including frequent software deliverable, daily meetings, daily interaction between business people and developers, continuous integration, no overtime, refactoring, and face-to-face team conversations.

Following the move it was clear that we had also improved our practice of collective code ownership while becoming better at moving people around.  We lost the benefits that come with face-to-face team conversations but overall our system had matured into a  production release and we signed our first customer.

The immediate impact following my move was surprising.  In these first few months, we experienced a team surge that saw us sprinting through 4 development iterations, invigorating the project and more importantly, introducing breakthrough features that singled out the product in the marketplace.

I believe this surge was directly related to positive effects that resulted from my move to a teleworking environment and I wonder whether the same intensity could have been achieved had the move not occurred.   The catalyst behind this surge included:

• Eliminating commute time gave me more time to focus on developing the product.

• The team raised its performance level in order to prove effectiveness in a virtual team setting.

• Three hour timezone difference allowed us to benefit from “Follow the Sun” development.

• Daily group chat sessions eased the recalcitrance and shyness that sometimes occurred in face-to-face conversations, empowering all members to speak up.  The result was continuous feedback that improved our team-intimacy, awareness to project risk and productivity.

Counter Balancing

After this initial surge and as the year progressed, some of the challenges facing teleworkers and virtual teams began to set in, specifically team intimacy began to suffer as the lack of face-to-face meetings, coupled with the transition of part of the team to separate projects began to disconnect and disorient us all.

To counter these effects, we:

• Introduced a weekly meeting with business leads.

• Transitioned my original role as tech lead to a colocated member, allowing me to better focus on clearing obstacles that were preventing colocated members from maximizing their productivity and brilliance.
• Increased my visits to the office.
• Decreased project velocity to achieve a more sustainable pace.

Subtle adjustments also helped counter the effects of my moving to a remote work environment.   For example, I could sense a collective relief from business leads (as well as increased call volumes) once my Argentine ring tone was changed to sound like the standard USA ring tone.

Conclusion

Looking back, I do believe the initial team surge was directly related to the benefits agile promotes in moving people around.   The intensity of this surge was largely a result of our response to the changing circumstances.   As challenges arose, it was critical to address each quickly and decisively.  In my new role as a teleworker, I can confirm the importance of continually reaching out to all stakeholders, make them feel comfortable that you are accessible during their work hours.  The move to a teleworking environment also gave me time to focus exclusively on writing code while transitioning my role as tech lead to a colocated member.  In the process we improved our practice of collective code ownership.

While this move served as an extreme case of the challenges and opportunities that result when you move people around, more traditional examples more would show team members owning different portions of the system or  fulfilling different team roles at different times in the development lifecycle.   When effectively managed, the benefits will be distributed across individuals, their team and the stakeholders they serve.

This concludes the series Moving People Around.

Throughout this past year I’ve presented various articles highlighting the high-performance computing requirements on Wall Street, where latency and temporal constraints are closely tied profit and compliance.

As many of you know, The Techdoer Times is also focused on topics surrounding highly-productive teams.   Our day-to-day experience with Agile software development makes us biased to the productivity benefits of this software development approach, especially with regards to Wall Street, where project success is rooted in a team’s ability to solve functional requirements, as is the case with most technology solutions, but also the ability to solve the complicated non-functional temporal requirements behind the industry’s performance and compliance needs.

We’re happy to share our nascent thoughts surrounding the cultural, managerial and engineering benefits of Agile on Wall Street at Agiles2008 – the first ever Latin American conference on Agile Development methodologies.

Our goals is to raise awareness to Agile for projects heavy in non-functional requirements.

A PDF formatted presentation can be downloaded here.   As always we welcome your comments or suggestions by emailing us at techdoer@gmail.com.

About 9  years ago I worked for the Food and Agriculture Organization, FAO, in Rome, Italy.   Our team was focused on creating technology solutions to enable sustainable agricultural development in developing countries.  Sounds great on paper, but the most valuable lesson was realizing the magnitude of the world’s hunger problem and how technology could help combat this problem.

Today is Blog Action Day.  Thousands of bloggers will unite to discuss a single issue – poverty.

In my previous post, I suggested that Agile’s success is rooted in the creative responses that result from embracing continuous feedback and satisfying the right customer needs.

In this third and final part to the Standing Up For Agile series, I’ll address Agile’s current criticisms and also take a look at the emerging challenges in software development.

The NaySayers

Since its inception, agile methodologies have been heavily criticized by industry practitioners.  Common complaints include the following:

• lack of discipline
• lack of focus towards non-functional requirements
• doesn’t work for with teams composed of junior-level people
• doesn’t support thorough architectural design and planning
• makes scheduling impossible

As a practitioner of agile methodologies, I can relate with these concerns.   The non-functional high-performance requirements on Wall Street represent the types of complex requirements that naysayers claim are not adequately addressed with agile software development efforts.  In addition, it is true that junior-level people lacking the experience and knowledge to make quick decisions in a constantly evolving process add significant risk to the overall result.

Perhaps the real challenge in succeeding with Agile, or any software development process, is understanding the true characteristics of the people, process and technology in the problem and solution domains to be able to tailor an approach that inherits the appropriate elements from the many software development methodologies.

Moving forward, Agile’s popularity over existing methodologies will continue to grow because agile practices are purposely designed to address the key differences between software development versus other engineering disciplines.  The same cannot be said for predictive methodologies like the Waterfall approach.

Standing Up For Agile

Agile acknowledges that many stakeholders don’t know what they want until they see what they don’t want, or see what they would like more of.

Frequent deliverables gives stakeholders something to build on.

Agile acknowledges that requirements change.

Test-driven development focuses junior-team members and provides regression testing capabilities for everyone.

Continuous integration synchronizes parallel efforts, minimizes the time-between-failures, and builds team moral and momentum by promoting an always buildable system.

A system metaphor serves as a figurative construct that unites stakeholders’ understanding of the problem and the solution behind it.

No overtime keeps team members productive, lucid and motivated.

Continuous customer interaction ensures the right customer needs are solved.

Agile, like Computer Science, acknowledges that complexity can be addressed with divide-and-conquer approaches by approaching development as a series of releases each consisting of multiple iterations.

Self-organization attracts the best programmers and project managers who tend to avoid command-and-control organizations.

Daily meetings promote accountability throughout the project’s life ensuring all team members carry their weight.

Refactoring addresses quality issues resulting from minimal planning and promotes frequent deliverables.

Pair programming drives up productivity and quality of code.

Collective code ownership promotes coding standards, improving code maintainability, facilitating moving people around and pair programming.

Looking Ahead

For better or worse, the reality is that more and more software development is occurring in virtual teams.  For Agile to continue to grow and mature it must successfully address the challenges of software development in these distributed settings.

This concludes our series Standing Up For Agile.  For more information please contact me at techdoer@gmail.com.

In part two of the Standing Up For Agile series, I’ll present the key elements behind Agile’s success in the early 21st century.

Thriving on Change

In some respects Agile methodologies represent an instance of Weinberg’s model.  Let me explain how.

Motivation

Constantly changing stakeholder requirements can certainly demotivate team members.  How does a software development methodology promote change while keeping managers, developers, and testers motivated?  The key seems to be in minimizing the time required to discover and react to change.  Agile methodologies promote:

• Daily Meetings – motivate through accountability
• Continuous Integration – synchronizes parallel efforts and  keeps the system buildable at all times
• Test-Driven Development – provides quick pulse on progress
• Frequent Customer Interaction – keeps everyone focused on the right set of needs
• No Overtime – keeps team members productive, lucid, and motivated.

These elements help mitigate the risk that progress will get too far off target, and more importantly that team progress is always visible and focused on addressing the true needs of the customer.  The same cannot be said for the heavyweight upfront planning that occurs in predictive methodologies like Waterfall or RUP, where the lack of frequent stakeholder feedback only increases the probability that the plan is not aligned with the most relevant needs of the customer.

Changing scope and requirements can demotivate team members if they conclude that a large portion of completed work was time wasted.  Agile prevents this by collapsing the time required to discover and adjust to change and instead motivates by ensuring team efforts are always addressing the most relevant needs of the customer, which only increases the chances for customer satisfaction.

Organization

Agile promotes small self-organized, self-empowered teams for good reason.  They represent and efficient and effective organization of disciplines that can best satisfy customer needs.  Agile represents a branch of Nonlinear Management and thus inherits similar criticisms including:

• Only works with senior-level developers
• Requires too much cultural change to adopt
• Can lead to more difficult contractual negotiations

These are legitimate concerns that reinforce the notion that implementing agile software development requires a lot of work.    Both Agile and nonlinear management implicitly tell us that in this era of knowledge work, where no hierarchy exists between managers and programmers, managers must adapt and evolve to promote and empower self-organizing teams within their organizations.

Innovation

Weinberg MOI model also suggests that innovation depends on the flow of ideas, understanding of the problem, and ensuring quality.   Agile’s promotion of daily meetings, frequent customer interaction, no overtime, pair-programming, moving people around, continuous integration and test-driven development all effectively address these aspects of innovation.

With daily meetings and moving people around, team members are continuously challenging and stretching their individual and collective understanding of the problem and solution domains.  With frequent customer interaction, the team members are educated on the subtle characteristics in the problem domain, knowledge typically locked in the heads of domain experts,  which increases the chances for an effective solution and overall customer satisfaction.  Promoting no overtime is a good way to keep team members lucid and motivated during productive hours.  Test-driven development promotes higher-quality deliverables by focusing developers and testers to the solution’s ability to satisfy requirements at any point during the project lifecycle.

With these elements in place, agile teams are empowered to continuously address the most relevant functional and non-functional requirements in most creative and effective way.

In part 3 to the series Standing Up For Agile, I’ll cover the criticism of agile methodologies as well as the emerging challenges in software development and how Agile can effectivley address them.