There are three benefits in adopting a behavior driven hiring process. First, with a focus on talents comes the advantages of a strengths-based hiring approach.  Second, when it comes to people, past performance may be the most accurate indicator of future performance.  Third, the understanding of talents that emerges from the hiring process, will help guide the self-organization phase for effective team growth after the candidate comes on board.

In this series Behavior Driven Hiring, I will show how a focus on recurring professional behaviors, during the hiring process, is the first step to building high-performance knowledge worker teams.

Please check back soon as I continue Behavior Driven Hiring.

I stumbled upon Bruce Tuckman’s, 5-stages of group development model (a.k.a. forming, storming, norming, performing, adjourning), which despite it’s age, and simplicity, proved timeless in helping me understand the maturity of the transient and core teams I was managing.   Fast-forward a few years, and my understanding of team dynamics has grown to include concepts that are more deliberate in describing the stages of growth for high-performing teams.  Although I liked Tuckman’s model in the past, I no longer find it useful because it does not make explicit the pre-requisites for effectiveness from one stage to the next.  For example, how well your team ‘performed’ depends on how well they ‘stormed’.  I want to remove any notion haphazardness in Tuckman’s model while making explicit the characteristics required for successful transition from one stage to the next.

Four Stages for Team Effectiveness

I’m proposing the following new model for effective team growth:

Self-organize -> Trust -> Adjust -> Deliver

The growing research and understanding of creativity, motivation and trust in knowledge worker teams inspired this Four Stages for Team Effectiveness model.  Derived from Tuckman’s model, it borrows concepts and terminology from Agile software development, the importance of trust in creative teams, and from Peter Drucker’s teachings on knowledge worker effectiveness.

Self-Organize

My first exposure to self-organizing teams, and the effectiveness self-organization brings, was in Agile’s Extreme Programming.  It therefore doesn’t surprise me to hear Dan Pink, The Surprising Truth on What Motivates Us, talking about the critical role autonomy plays in fostering motivation in creative types.  The link between autonomy and self-organization is how they work hand-in-hand towards ensuring the most effective team members are assigned to productive work as well as inviting the cohesion and commitment required to overcome big challenges.   Command-and-control organizations lacking this autonomy and self-organization across their structure, are forced to rely on a complex and potentially demotivating system of closely monitoring people, and their productivity, while providing the “carrot and stick” rewards to keep them going.

Trust

By trust I not only mean the trust between team members, but also the trust between a team and their stakeholders, their customers, their organization, their processes and the tools they use to build solutions.  Diana Larsen’s Agiles 2009 presentation on trust describes in more detail the necessary components of this trust:

Adjust

I write this at the conclusion of an intense stage of adjustment within my own team. Looking back, I realize the solid foundation of trust we built leading up to our adjustment stage has helped us survive the tension and discomfort that resulted from, as Jim Collins wrote in his book “Good to Great”, confronting the brutal facts.  No matter how much you’ve grown to like and trust your team, uncomfortable periods of adjustment are inevitable.  In our situation, the primary catalyst for adjustment has been the gap between what we should be doing, versus what we were doing.  Your own team’s awareness and intolerance to these types of gaps will also increase as your effectiveness grows.

Deliver

High-performance and effectiveness will be the signature of teams who have grown and mastered these previous stages. By this stage they will be an autonomous, cohesive group, committed to doing the right things while delivering immense value and equally important, using the feedback loop that emerges from their performance to reinforce and grow their foundation of trust and effectiveness.

Conclusion

Think back to the software development teams you’ve been a member of.  Do you remember your own stages of forming, storming, norming, performing or adjourning?   Now compare the stages of growth between a lackluster team you worked with and a high-performing team.   How did the stages of growth differ between these teams?   I’m proposing a new model for growing effective teams and the elements of this model are rooted in exactly those differences.

Share your own stories of team growth and effectiveness by emailing techdoer@gmail.com.

Click to see the previous day blogs:

Monday, June 7, 2010
Tuesday, June 8, 2010
Wednesday, June 9, 2010
Thursday, June 10, 2010

All Railsconf 2010 keynotes on youtube available here.

As I mentioned previously, this struggle does not belong to entrepreneurs alone.  Knowledge workers also need to continuously align with their strengths, at the same time effectively growing their knowledge and skill, if they are to make a remarkable impression on their customers.  In this final post I’ll talk about the other side of the struggle, that of finding or creating the right opportunity.

Quick Rewind

Thus far I’ve stressed the importance of knowing and building on your strengths, as well as becoming more effective in both thought and action.  I believe these are essential ingredients in forming the foundation of autonomy, mastery and purpose that is so important to the motivation of creative people.  As I conclude this series, a narrative has emerged and it reads as follows:

Achieving and growing this level of effectiveness will permit you to grow your competence in a unique and powerful way.  This results in elevated levels of motivation which further amplifies your sense of purpose, autonomy and mastery, which further increases the number of opportunities you encounter and your ability to succeed in them.

In essence I’m saying that the number of opportunities you encounter will depend on your level of effectiveness.

What is Opportunity?

Look up the definition of ‘opportunity’ and you’ll find words ranging from ‘possibility’ to ‘chance’.  For entrepreneurs, an opportunity is a calling to act, and every aspect of this ‘calling’ will directly relate to the effectiveness of the person who found it.

I asked my brother, a successful designer and woodworking entrepreneur, how he finds opportunity:

“The right opportunities come from surrounding yourself with the right people, this includes prospective clients.  Imagine you are an explorer or hunter entering a forest.  Not just any forest, but the right forest. This is your ‘market’.   The machete and other tools you carry will help clear the mess and pave your path forward.  Preparation means sustaining and growing the effectiveness of your mind, body and machete. This is what you can control.  In this context, the longer you search, the greater your chance of finding the reward. And don’t forget to enjoy and refine this process throughout, it will increase the chance of future reward.”

It’s Simple Really

As a person who creates and does things, the way to find and create new opportunities is to continuously align with your strengths while growing your ability to decide and do the right thing across the spectrum of people, processes and technology/tools.  I’ll end this series with a relevant quote from Guy Kawasaki that says “Make meaning and you’ll make money”, perhaps better restated as find meaning and you’ll make money.

I hope you enjoyed this series on the Entrepreneurial Spirit.

For questions for comments please email techdoer@gmail.com

The Effective Executive

In part-1, I suggested that one of the many keys to the success of software company Balsamiq, was the methodical preparation of its CEO Giacomo ‘Peldi’ Guilizzoni. Luck requires preparedness and his preparation not only included 10,000 hours of professional programming practice, but the acquisition of management and leadership skills while still working for Adobe. During his presentation at the 2010 Better Software Conference in Florence Italy, it was easy to see how these new skills coupled with his innate charisma and humbleness might have served him well as an effective leader and first employee of his new venture.

Peter Drucker taught us the following traits of the effective executive:

1. Executives have to know where their time is being spent.
2. They must focus on outward contributions: on results rather than work.
3. Build on strengths first, and then give attention to areas of weakness.
4. Concentrate on the few major areas where superior performance will produce outstanding results
5. Make effective decisions.

Browsing through Peldi’s online presentations and interviews, you’ll see references to the  precise timeline of events and key milestones in the life of his company.  You will also detect his conviction when deciding what NOT to focus on, especially during the early days when risk of failure was high.  Examples like these are in line with the traits described above.  Knowing your strengths and where you excel, for example, provides a start to effective decisions making.  It will also provide the knowledge that helps you find those things you enjoy doing.

So how can you go about discovering your strengths?

Emotional Intelligence

As Marcus Buckingham and Curt Coffman so eloquently state in their book, First, Break All the Rules, the equation of competence includes the set of talents, skills and knowledge one possesses.  Surely a successful leader will have learned to grow his competence, but this can only be effective in the context of the intimate understanding he has of his own strengths.

To begin understanding your own strengths, it’s helpful to think about how others perceive you.  How were you praised or criticized in previous performance reviews? What suggestion for improvement were you given?  In a previously stressful situation, how did you react?  Answers to these questions will provide the clues you need to get started.  I also recommend reading through the panorama of talents, as well as completing Gallup’s StrengthsFinder assessment.

Identifying and acting on your strengths allows you to engage in a symbiotic and efficient system of effective preparation, which leads to the right opportunities, which in turn leads to more effective preparation and so on. This is how you amplify your strengths.

We are all Executives

A key point I’d like to introduce here is that the concept of effective executive not only applies to entrepreneurs, but to all knowledge workers, from generalists to specialists, across the organizational hierarchy, programmers included.   We live in a world where our success as knowledge workers, as well as the success of our team and organization, no matter it’s size, depends on an ability decide and execute ‘the right thing’.

Getting Back to Luck

Allow me to be overly simplistic for a moment.  We all agree you have to be lucky to succeed as an entrepreneur.  So if luck is the product of opportunity and preparedness, and preparedness involves the 10,000 hours of practice mentioned by Gladwell,  the understanding of strengths, skills and knowledge suggested by Buckingham and Coffman then how is one supposed to create the right opportunities?

Click here to continue to part-3.

The most beautiful thing we can experience is the mysterious. It is the source of all true art and science. – Einestein

What this also implies, which you can confirm by listening to their presentations or reading their blogs, is the existence of a struggle that sees them continuously aligning with a deeper purpose while amplifying talents in order to transform the lives of their customers.  This struggle is similar to the Hedgehog concept presented in Jim Collin’s book Good to Great and one that marketing has detoured around for the last 20 years.  In this post, Entrepreneurial Spirit, I’ll present the characteristics of this unquenchable struggle that drives these entrepreneurs towards successful software ventures.

Luck of the Italian

We’ve all heard the expression “Luck is when opportunity meets preparedness” and it’s hard not to find a whole lot of this luck in stories like that of a rapidly growing and successful software startup, Balsamiq, and its young founder and CEO, Giacomo ‘Peldi’ Guilizzoni.   I recently had the opportunity to listen to the charismatic and passionate CEO during his presentation at the 2010 Better Software Conference in Florence, Italy.  Throughout the presentation, it struck me that yes, his company has had its fair share of luck, but only because of his remarkable pursuit of preparation and opportunity.

During the presentation, he talked about his persistent desire to start his own software company but also pointed to a phase in his life when he simply wasn’t ready to commit to this venture.  The story goes that in 2004, and with a few years of professional software development experience under his belt at Adobe, he picked up a book from Barry Moltz, You Need to Be a Little Crazy: The Truth about Starting and Growing Your Business, only to put the book down “after 20 pages”, admitting to overwhelming reluctance to absorb the risk needed for such a venture.   If he admitted to a lack of preparation in 2004, what changed by 2007, the year he committed to and officially launched Balsamiq?

Power of Time

In Malcolm Gladwell’s book, Outliers, he talks about the 10,000-hour rule, or the minimum time of practice required for success in any field.   Peldi wasn’t prepared to take the risk to start his venture in 2004, but by 2007,  he not only completed his 10,000 hours of professional practice but had acquired the broader knowledge and skills of an effective leader.  Borrowing the words of the legendery Peter Drucker,  during these three years Peldi dramatically matured his ability to answer the question, “What is the right thing to do?”, he was on his way to becoming the effective executive.

Part-2 to the Entrepreneurial Spirit continues here.

Programmers and startups have also found themselves vying to become the next Bill Gates, or creating the next Google, but rarely do we sufficiently understand the personal characteristics that inspire such distinction.  For example, we know the benefits behind Agile’s iterative approach,  or model-view controller in Ruby on Rails, but do we agree on the interpersonal traits required for a high-performing individual?  In a world where software development processes, and technologies continue their trend towards commoditization, it is the ability to answer this question, and others like, that will begin to differentiate an organization in the marketplace.

People Narrative is a look at the traits that distinguish talented people in the software development lifecycle.  A comprehensive panorama of such traits can serve as a useful reference for programmers, project managers or executives to better understand, select, and improve the people in their own software development teams.

Competence

What talents come to mind when you think of the programmers you’ve worked with?   Was it their strong ambition, power of persuasion, or their ‘seeing eye’ intuition that set them apart?   Or maybe it was their relentless attention to detail coupled with an insatiable curiosity that drove their success.   Like them, you also possess innate abilities that uniquely (un)qualify you for certain types of work. Your ability to perform at a high level will largely depend on how well the unique talents you possess align with the needs of the job.

Still too many companies don’t understand this. They write job postings emphasizing a particular skill or body of knowledge yet neglect to mention specific talents required by this job. As Marcus Buckingham and Curt Coffman so eloquently state in their book, First, Break All the Rules, it is critical for companies to understand that the equation of competence includes the set of talents, skills and knowledge an individual possesses.   The key differentiator between these three is that you cannot teach talent.  For example, I’m willing to bet that an innate attention to detail was a major contributor to the performance of the last superstar programmer you worked with, or that good listening skills contributed to the excellent communication in your current team, but only a healthy dose of empathy has helped you fully understand them (i.e. Peter Drucker’s “the most important thing in communication is hearing what isn’t said”).

The performance of your software development teams will primarily depend on the individual and collective talents that characterize them.    I’ve included a comprehensive list of these talents in Table 1.

Intelligence
To better understand this list, I’ll need to categorize.  In their book,  Buckingham and Coffman offered striving, thinking and relating as three possibilities.  Using their scheme, striving talents such as  determination would explain the ‘why’ we perform, thinking talents such as a superior analytical ability would explain ‘how’ we perform, and finally relating talents such as charisma would explain for ‘whom’ we best perform.

The Theory of  Multiple Intelligences, proposed by Howard  Gardner, and more specifically the categories of intelligence he defines, may also provide a classification.   Howard suggests eight different types of intelligence including:

• Bodily-kinesthetic
• Interpersonal* (i.e. social intelligence)
• Verbal-linguistic*
• Logical-mathematical*
• Intrapersonal* (i.e. emotional intelligence)
• Visual-spatial*
• Musical
• Naturalistic

I’d like to assume that the distinguishing talents of high-performing software development teams span those intelligences marked with an asterisk.   More important is the understanding that successful professionals and teams possess talents not typically understood or revered by hiring organizations or educational institutions.  For example, standardized intelligence tests that limit themselves to measuring logical-mathematical and verbal-linguistic abilities.

Effectiveness

In software development terms, think of effectiveness as the runtime engine.  When the code is syntactically correct, it can be executed.  Every programmer can admit to an overwhelming sense of motivation that comes with seeing their code execute flawlessly. But without this runtime engine, even the most beautiful piece of code is rendered functionally useless.

Similarly in people, maximizing effectiveness requires a thorough understanding of their competence so that productive work can be assigned to the right individual.   While it may be straightforward to assign programming work to an individual who stands out for his knowledge of Java, it may not be so straightforward to conclude that the work of a leader should be assigned to an individual with a talent for empathy.   When the full spectrum of an individual’s competence is required for the work at hand, the individual’s motivation will grow, and she will be satisfied in her job.  There has been strong emphasis on aligning business and IT over the years, and it’s time we also realize the impact of (im)properly aligning a person’s competence with their work.

Lacking a thorough understanding of an individual’s competence is a recipe for their ineffectiveness.  Peter Drucker taught us that the job of the knowledge worker is to be effective (and similarly stay motivated) while also reminding us that:

“…people of high effectiveness are conspicuous by their absence in knowledge jobs. High intelligence is common enough amongst knowledge workers. Imagination is far from rare. The level of knowledge tends to be high. But there seems to be little correlation between a man’s effectiveness and his intelligence, his imagination, or his knowledge. Brilliant men are often strikingly ineffectual; they fail to realize that the brilliant insight is not by itself achievement.”

Perhaps he referred to the limited definition of intelligence and brilliance, which is to say, achievement requires the broader set of intelligences usually found in more than one person alone.   Our teams should be formed on this basis and as Marcus Buckingham and Curt Coffman stated, the job of a manager is to convert these combined talents into performance.

Figure 1.  High-Performance Teams

Trust

Diana Larsen’s presentation on Trust taught us that an individual’s competence (i.e. talent, skill, knowledge) is a major component towards growing professional trust  (See Figure 1).  A sufficient level of trust must exist in order for a self-organizing team to maximize their individual and collective effectiveness.   It will serve as the foundation for the team’s understanding of each individual’s competence, of how best to organize, as well as maximizing their ability to execute.

Conclusion

It should be clear that before we can preach the benefits of Agile software development, or evangelize the next great programming language, for example, we need to understand the talents of people, because it is these talents, and our ability to nurture them, that will primarily influence the foundation for individual and team trust, effectiveness, motivation and overall performance.

But what about testers…

I know of a team in which every member is called a programmer, not a developer or a team member. Do you believe it? It is true, but isn’t that strange?

What if said “I know of a team in which every member is called a tester”. Would you believe it?

Brian Marick was one of the original authors of the Agile Manifesto and the lone self-defined tester in the group.  He also made big contributions to the redefinition of our role, and others follow the path, but we have had what I consider (re)foundational book, Agile Testing, for less than 6 months now.

We are still in transition as a community . This transition is sometimes caused by a lack of understanding of Agile Development and sometime it is caused by the lack of breakthrough innovations.  There is some re-thinking going on, both in the broader sense (see Brian Marick on Micro-Scale Retro-Futurist Anarcho-Syndicalism), and in a more restricted sense (there is a group, AA-FTT – Agile Alliance Functional Test Tools, that is trying to advance the state of the art, you can see the discussions on the mailing list and a there are a number of workshops [1][2]).

So it might be too early to have a solution to the tester’s dilemma, but anyway, we have to continue to test and deliver products today. What is the best we can do right now?

Let’s discuss a promising proposition: BDD and Cucumber, as presented in The RSpec book: Behaviour Driven Development with RSpec, Cucumber and friends.

But first we need a little of history and background.

Test driven development (TDD)

TDD was a wonderful idea that came to us in the XP package. It is widely accepted as a good practice but isn’t so widely used.  Why?  TDD had the positive effect of pulling up quality, both as a concern of the team and in the product result. But the four letter word (test) caused a lot of friction and misunderstanding.  The friction often appears as a resistance of programmers to do “tester’s job”.  The misunderstanding could be caused by the way people learn new concepts. If the words are known, they usually assume that the concept is the same (see Sapir-Whorf Hypothesis).

That’s why, instead of Agile Development, Brian turned to Micro-Scale Retro-Futurist Anarcho-Syndicalism (reading that, you most certainly won’t have a clue what he is talking about, so you will go and check) and Kent Beck turned to Responsible Development (he prefers the connotation of responsible; being responsible has some cost attached; being agile, well … who doesn’t want to be agile?)  As a result of getting TDD wrong, some practitioners focus on testing the code, while they should be paying attention to the design. This loss of focus leads to bad practices; for example, when a developer defines tests that depends on the implementation (internal structure) instead of the behaviour of the object.

Acceptance Test Driven Development (ATDD) …

You might like to know that software development and theater companies have many things in common: “Good theater companies routinely produce innovative products under extreme deadline pressure (opening night) ”. Or at least, they could have. To get those results (reliable innovation), you have to build products through what Rob Austin and Lee Devin called Artful Making, instead of Industrial Making. This is not a case of wishful thinking, you need some precondition to use Artful Making. One of them: Quick, low cost and high quality iteration, and a product at the end of each iteration (yes, this sounds very Lean).

One impediment to Artful Making is the language/conceptual gap between the user and the development team. If we have different language, we incur in translation time and cost and we risk to inject error every time that we cross the gap… in any direction. For instance, during requirement elicitation and during acceptance test.

Would it be nice if we could have requirement stored in such a way that:

• They are understandable for both the users and the development team (communication).
• They could be defined (written) before the product is build (test).
• The product could be quickly and easily (automatically) validated against them (test).

Enter Fit: Framework for Integrated Test. What if we let the user write the requirements as examples tables in Word, Excel or HTML pages? And then, with a minimum gluing code and a test runner, we could test the application.

It was a great leap forward, both in communication and test. It has been followed by FitLibrary and Fitnesse which add a wiki as the front end (instead of HTML).

A lot of agile teams use some incarnation of FIT/FitLibrary/Fitnesse.  Some problems where reported:

• Many end user don’t take ownership of the tests. Team members (usually the testers) act as scribes, and translate user / team conversation to HTML /wiki.
• When the number of test cases scale up, test case development become awkward. A wiki or a set of html pages are not a good development environment. As a developer, you have to manage two models of thinking (FIT/Fitnesse and your application programming language and environment)

…And Behaviour Driven Development (BDD)

BDD is an evolution of TDD and ATDD proposed by Dan North (video).

We can get rid of Test in TDD, and call it Behaviour Driven Development. But why Behaviour? Because it is better to focus the programmer in the more important aspect of the objects, their behaviour. It is also built upon the idea of a ubiquitous language as in Domain Driven Design.

To complete the name/concept switch away of Test, the executable specification are called examples. So specifications of code behaviour are called code examples and the higher level specification are called application examples.

At the requirement/acceptance level, we define application behaviour, at the design/programming level, we define object behaviour. In both cases you follow the red/green/refactor cycle, having two, concentric cycles. The outer cycle is the ATDD part, the inner cyle is the TDD part, but the both work together. These are the steps, number between () refer to the figure extracted from The RSpec Book from The Pragmatic Bookshelf.

1. Write an scenario with Cucumber (1)
2. Write a step definition
3. Run and Watch it fail (2)
   1. Write a failing code example with RSpec (3)
   2. Get the example to pass (4)
   3. Refactor (5)
   4. Repeat 3.1 – 3.3 until Cucumber step is passing
   5. Repeat 2 – 3 until Cucumber scenario is passing (6)
   6. Refactor (7)

Building the application in this way promotes an Outside-in development. Models appear as solutions to the acceptance criteria. This has a few implications: it is easier for the client to write acceptance test (as examples), the model is more relevant to everyone on the team, and it requires the use of techniques to support incremental development (like mocks).  The bottom line of BDD is “writing software that matters”, and it follows three principles:

• It’s all behaviour. This behaviour should be described in a common (ubiquitous) language, that can be understandable by the business and technical people on the team.
• Deliver stockholder value. Brian Marick will say “focus on the cheese”.
• Enough is enough. This is true for upfront planning, design, etc, but also for automation, don’t try to automate everything.

There are many tools implementing BDD ideas with different approaches and contexts (i.e. languages). In Ruby, the tool stack includes RSpec, Cucumber, Webrat. I will only drill into Cucumber which is the BDD tool most relevant to end user and (non technical) testers.

Cucumber

I will not try to explain the use of Cucumber. Following the excellent Ruby/Rails tradition of web/podcast you can check some Cucumber resources here and of course, check the Cucumber site and The RSpec Book.

Disclaimer: Please consider that I am reading a beta version of The RSpec Book, a nice feature of Pragmatic Bookshelf, and I haven’t research Cucumber as much as I would like, so I might be missing many features. I just want to share some impressions as a newbie user.

Just to agree on vocabulary, this is a short example (login.feature):

Each feature file contains:

Feature: Login
In order to use the site functionality
As a guest user
I want log in

Scenario: I try to access without logging in
Given I am not logged in
When I go to the taskboard number 1
Then I should be on the login screen

• One feature, with a free description (not parsed), that could take traditional story format (As… I want … So that I can …) or another format that change the order to put more emphasis on the value obtained (In order to… As a … I want …). The last format is closer to the order of the steps.

• One or more Scenario, each scenario has a name and a number of steps. Each step begins with any of five keywords: Given, When, Then, And, and But.
  • Given: define the context, the preconditions.
  • When: is the action or event that happen to the application.
  • Then: is the expected result.
  • And/But: could be used after Given/When/Then to add steps with the same meaning of the previous one.

Feature files are written in a very simple DSL (Feature/Scenario/Given/When/Then/And/But) called Gherkin. Like Python and YAML, Gherkin is a line-oriented language that uses indentation to define structure.

My first impression, coming from FIT/Fitnesse, is that it is not so different, I mean, you can also have a Do fixture that can be read almost as English. Almost.

Examples written in natural language (like English, but many languages are supported) is a big selling point. And you don’t lose the table (data driven) approach (Scenario Outlines).

It is far easier to edit, just plain text and it seems to be more integrated to the development environment. I really like the ability to switch more easily between application examples (feature file) and code.

Working in a Ruby environment also allows a more quick and simple cycle that I think could help when pairing with a user.

It’s sad to say that I don’t have experience with large set of features. One of the problems with FIT/Fitnesse is that it is more difficult to refactor examples than to refactor code. And sometimes, you unwillingly break the examples by changing an apparently innocuous html. I don’t know how scaling and refactoring is managed in Cucumber (except the Scenario Outlines).

One downside of plain text is the loss of styling and hyperlinking. The first one could be replaced by convention over configuration and some viewer, but without hyperlinking, we lose the ability to arrange examples as a nice snippet of documentation.

Final comments

In the BDD double cycle, you can extend the use of Cucumber to make code examples or extend RSpec to make application examples.

The questions that come to my mind is, why developers will not choose to use RSpec, so they keep the toolset simpler. Anyway, they don’t need English if they can read Ruby.

In my experience, the cost to maintain the gluing code is only acceptable when there are full time team members that are not fluent in the main programming language (say Java or Ruby). It could be the user or somebody working as the user scribe (an Analyst or Tester).

I see that Cucumber has many good point, and a superficial use is not enough to judge it. I would like to give it a deeper try. Do you have a project Rails? Call me

Will testers role in agile team become the user scribes of functional tests? Yes, probably. But not only that. Exploratory testing is much needed, and a devil’s advocate is useful for the team. For instance, related to coverage, test automation and risk analysis.  The Agilar Taskboard team (Francisco Tufró & Marcos Navarro) allows me to use the product as a test bed for Cucumber.  Thanks!

About the author: Juan Gabardini has more than 20 years of experience in IT management, IT product development, teaching and consulting in financial services, retail, telecommunications, medical equipment and manufacturing sectors.

Juan is currently focused on testing and coaching the Agile way, and building an Argentinian and Latam Agile community (Agile Open Tour and Ágiles 200x).

Juan is a member of IEEE, Agile Alliance and Scrum Alliance, and holds degrees in System Analysis and Electronic Engineering from the Buenos Aires University, where he also is teaching. You can find him (in Spanish) in http://softwareagil.blogspot.com and http://www.agiles.org

The Programming Languages

When language architects were designing the foundations of the most popular programming languages, multi-core microprocessors were hidden in laboratories. Only high performance servers had access to multiprocessing systems. Just a few specialized workstations had more than one CPU installed. Therefore, C# and Java offered support for concurrency and multi-threading intended to offer more responsive applications. However, language architects didn’t design this support to optimize applications for future multi-core CPUs. Hence, nowadays, it is really difficult to optimize existing code to take advantage of multi-core CPUs using frameworks prepared for serial code.

In order to take full advantage of multi-core, the applications need a task-based design and a task-based programming. There is no silver bullet. So far, there is no way to optimize an application recompiling it without changes. Some developers expected this to happen. There is a great need for new designs, new programming techniques and new tools. The software development industry needs a new great paradigm shift.

Besides, developers need a framework capable of handling tasks. There are new programming languages, or new versions of older concepts, like functional programming. Functional programming makes it easier to code task-based designs and to split the work to be done in multiple independent tasks that could be run in parallel on multi-core CPUs.

There are many new programming languages with a great focus on functional programming, prepared to take full advantage of multi-core and to offer a great scalability. Just to mention a few:

• Scala
• Haskell. Yes, the one that has more than twenty years. Pure functional programming languages are back and they can be the future for parallel programming.
• Microsoft Axum (formerly Maestro).
• Microsoft F#.

However, do developers want to begin learning new programming languages? Most developers want to leverage their existing knowledge to move onto multi-core programming.

C++ and Fortran programmers had early access to parallel programming. Nowadays, these are the only programming languages that can take advantage of the full power offered by modern microprocessors. C++ is closer to the hardware. Hence, it allows many optimizations that aren’t available in any other programming language – apart from C and assembler. You can access all the vectorization capabilities from C++ and Fortran.

OpenMP has been offering a high quality multi-platform and open source shared-memory parallel programming API to C/C++ and Fortran for many years now. Besides, Intel Threading Building Blocks, also known as TBB, allows developers to express parallelism in C++ applications to take advantage of multi-core.

Message Passing Interface (MPI), is a language-independent communications protocol used to program parallel computers. You can use MPI to take advantage of multi-core on many programming languages. However, its main focus is to help develop applications to run on clusters and high performance computers.

Intel offers compilers and libraries optimized to take advantage of multi-core. However, you still require coding your applications considering new parallel designs. The usage of vectorization in their math libraries is a great plus point. There is an outstanding opportunity for new libraries and components optimized for multi-core and vectorization. Parallelism brings new opportunities to the software development industry.

There are new companies taking advantage of the need for multi-core optimizations, like Cilk Arts, offering its Cilk++ compiler. It is based on GCC and includes a modified compiler and debugger to simplify multi-core programming for Linux and Windows platforms.

Mac OS X’s Xcode development environment offers access to Grand Central Dispatch and OpenCL. OpenCL allows C programs to run in the GPU instead of loading the main CPU. The interest of developers in Xcode has really grown since multi-core and OpenCL.

C# and Java are evolving in order to offer developers new ways of expressing parallelism in their code. Indeed, they are changing many aspects that were designed for another world, the old single-core machines.  Some of these changes include new garbage collectors, new frameworks and features, new functional approaches and task-based programming capabilities.

Java 7 will offer the new fork-join framework, really optimized for multi-core.

C# 4.0 (Visual Studio 2010) will add task-based programming capabilities and parallelized LINQ (PLINQ). Besides, it will allow the possibility to manage the desired degrees of parallelism.

Furthermore, there are new DSLs (Domain Specific Languages) to add parallel programming capabilities to existing high-level languages. For example, GParallelizer adds nice parallel programming capabilities to Groovy.

Most modern programming languages are evolving or adding inter-operatibility capabilities with other languages to favor multi-core programming.

However, don’t forget about vectorization. Mono, a free and open source .Net compiler, offers access to SSE3 and SSE4 for C# developers.

A few years ago, concurrency was about threads. Now, experts are talking about tasks and fibers. Why? Because in order to develop an application using a task based approach, threads are too heavy. Tasks and fibers are lightweight concurrency elements, much lighter than threads. They allow developers to implement complex task-based designs without the complexities of threads.

Ruby 1.9 added fibers and it simplify the creation of pipelines. Pipelines take great advantage of Hyper-Threading combined with multi-core.

If you want to go parallel, follow James Reinders’ eight key rules for multi-core programming. You can apply them to any combination of programming language, framework, compiler and virtual machine.

Java 7 and .Net 4 will not offer framework support for vectorization (SIMD instructions). SIMD has been there since Pentium MMX microprocessor.  This decision doesn’t make sense especially considering that there is a huge market for smart professionals in the new parallelism age.

The New Tools

A well-known proverb says “A good workman is known by his tools”.

A parallelized application requires new debugging and testing techniques. You need to catch potential bugs introduced by concurrency.

Intel has been offering tools for High Performance Computing and parallelism for many years now. A few weeks ago, Intel launched one of the most complete parallel toolkits for C/C++ developers, Intel Parallel Studio. Among many other features, it helps developers to compile applications tuned for multi-core CPUs and to find concurrency specific bugs and bottlenecks. You should expect to see more tools like this coming on the next quarters.

Visual Studio 2010 will add enhanced multi-monitor support capabilities. You’ll need more than one monitor in order to debug applications running with a task-based approach. It will also add task-based debugging capabilities. However, Visual Studio 2010 has recently entered Beta 1. Therefore, if you want to develop an application using a task-based approach using C# 3.0, you still have to work with threads. Visual Studio 2008 offers nice multithreading debugging capabilities.

Most IDEs are changing to offer new task-based programming, debugging and testing capabilities. You have to test parallelized applications on multi-core CPUs. Many bugs aren’t going to appear when running them on single-core CPUs.

There are many free tools to help you in the multi-core jungle. You can monitor your applications and test their concurrency efficiency using Process Explorer and Intel Concurrency Checker.  If you use these tools to check commercial software, you’ll be able to see the need for new multi-core aware developers. Besides, you’ll see a lot of opportunities in the multi-core age.

By the way, multi-core programming has a high quality weekly talk show lead by Intel experts, Parallel Programming Talk.

Summary

While the old free lunch is over, the industry is reshaping itself to take advantage of the new microprocessors architectures of today and tomorrow.

Hardware will continue to advance and offer more parallel processing capabilities, even though the software industry is moving more slowly than expected.

Bottom line is that multi-core seems to be a really sustainable competitive advantage that requires a great paradigm shift from developers throughout the software lifecycle. There is light at the end of the tunnel. Are you ready to reach it?

About the author: Gaston Hillar has more than 15 years of experience in IT consulting, IT product development, IT management, embedded systems and computer electronics. He is actively researching about parallel programming, multiprocessor and multicore since 1997. He is the author of more than 40 books about computer science and electronics.

Gaston is currently focused on tackling the multicore revolution, researching about new technologies and working as an independent IT consultant, and a freelance author. He contributes with Dr Dobb’s Parallel Programming Portal, http://www.go-parallel and is a guest blogger at Intel Software Network http://software.intel.com.

Gaston holds a Bachelor degree in Computer Science and an MBA.You can find him in http://csharpmulticore.blogspot.com and http://software.intel.com/en-us/profile/417051/

The Operating Systems

No matter the version, Mac OS always had a great advantage over any other desktop operating system. It knows exactly the underlying hardware because it is designed for running on Mac’s certified hardware only. You can run it on different hardware, at your own risk.  The same company develops the computers and the operating system. Leaving its great innovation aside, this is its great secret. For this reason, it can be tuned to take full advantage of specific hardware. For example, Mac OS X’s latest versions running over Intel microprocessors take advantage of vectorization, they use SSE (Streaming SIMD Extensions) and SSE2. In fact, Apple has been promoting vectorization and SIMD (Single Instruction Multiple Data) instructions in its Developer Connection Website.

However, Mac OS X Snow Leopard is giving another great step, offering Grand Central Dispatch.  Nonetheless, there is a big problem. There are too few developers working with specific Mac developer tools. Mac is also going to the 64-bits arena.

FreeBSD is one of the free and open source operating systems that always offered great features when working with multiprocessor systems. FreeBSD 7 works great with multi-core CPUs as well. Therefore, many high-performance servers around the world trust in FreeBSD’s scheduler.

The key is the operating system scheduler.  It is responsible for distributing the physical and logical processing cores, and assigning processing time to each concurrent thread.  It performs a really complex task.  For example, an operating system running over an Intel Core i7 CPU has eight logical processing cores (it can run eight hardware threads concurrently), but four physical cores. It has to distribute dozens of threads on time-slices available from eight logical cores. Thus, the scheduler efficiency impacts on the application’s performance. An inefficient scheduler could ruin a highly parallelized application’s performance.

Linux works great with multi-core CPUs, FreeBSD works better , but Linux does a great job. However, many desktop applications running on Linux GUIs aren’t optimized to take full advantage of multi-core. Nevertheless, Linux running as a Web server in a classic LAMP configuration is really fined tuned for multi-core.  Free Linux versions have a great advantage on the multi-core world as they don’t limit the number of cores using expensive licenses. Therefore, you can upgrade your desktop or server without needing to worry about the number of cores supported by your operating system license.

Both FreeBSD and Linux have a great advantage over Mac and Windows, most new deployments using these operating systems are using the 64-bits versions. Applications running in 64-bits offer more scalability than their 32-bits counterparts.  Parallel algorithms require more memory than serial ones. Most operating systems running in 32-bits can only address 4 GiB. There are some techniques to work with more memory even in 32-bits. However, they reduce the memory access performance.

4 GiB could seem a lot of memory. Nevertheless, as the memory map is a bit complex, you need some spaces for other purposes, like writing data to the video memory and other buffers. Hence, the operating system cannot access the whole 4 GiB of main memory. Furthermore, some operating systems limit the maximum memory addressable by an application in 32-bits mode (2 GiB in Windows standard configurations). Again, there are some techniques to work with more memory for the applications.

Whilst working with large images, videos and databases, these limits could be a great problem to add scalability. More cores mean more processes, more threads, more tasks or more fibers. A 2 GiB memory limit for a single process could mean a great I/O bottleneck.

Hence, working with 64-bits is a great step ahead. You have the same driver problems in both 32-bits and 64-bits Linux. Thus, you can install 64-bits Linux without worrying about additional problems. Working with 64-bits, you can scale as the number of cores increase without worrying about memory problems. Of course, you do have to worry about available physical memory. However, that’s another problem.

FreeBSD and Linux schedulers already offer an excellent support for NUMA. However, you may see nice improvements in future kernel versions. The idea is simple, the more the number of cores, the more optimizations required for an efficient scheduler.

Now, let’s move to Windows wonderland. Windows has a great disadvantage; each version offers different capabilities related to multi-core. You should check the maximum number of cores and the maximum memory supported by each Windows version. This doesn’t mean that Windows newest versions aren’t capable of handling multi-core CPUs. The problem is that they have different licenses and prices.

If you want scalability, you shouldn’t consider 32-bits versions anymore. Windows 2008 Server R2 and Windows 7 will support up to 256 logical processor cores. Nowadays, 256 logical cores seems a huge number. However, an Intel Core i7 is a quad-core CPU, but it offers eight logical cores (2 logical cores per physical core, 2 x 4 = 8). New CPUs with 8 physical cores are around the corner. Besides, they will offer 16 logical cores (2 logical cores per physical core, 2 x 8 = 16). Hence, you will see 16 graphs in your CPU activity monitor. An operating system offering support up to 256 logical cores really makes sense for the forthcoming years.

Despite being criticized everywhere, Windows Vista offered nice scheduler optimizations for multi-core. Windows 2008 Server R2 and Windows 7 will offer support for NUMA in their 64-bits versions. However, you must use the new functions to take full advantage of NUMA capabilities.

No matter the operating system, the applications must be prepared to take advantage of multi-core CPUs.
The operating systems are adding multi-core optimizations. However, except from Mac OS X, most applications running on the operating system take advantage of neither multi-core nor vectorization. I really cannot understand this.

I do believe it is time to learn from Mac OS X. For example, Windows 7 should offer a special version, let’s call it 7 Duo. 7 Duo could require at least a dual-core CPU with SSE2. So, you wouldn’t be able to run 7 Duo on older machines. If you have newer hardware, you’d buy and install 7 Duo. The operating system would load much faser, taking full advantage of your modern hardware. Your favorite Web browser should take advantage of multi-core and vectorization when parsing HTML or XML. Check this white paper, Parallelizing the Web Browser.

The great problem with PCs (x86 family) is the backward compatibility. It’s been an incredible advantage over the last decades. However, it is time to take advantage of modern hardware establishing baselines.
The same happens with Linux. I’d love to install an Ubuntu Duo on my notebook.

The operating systems are really crucial to tackle the multi-core revolution. Their schedulers are very important to transform multi-core power into application performance. Vectorization and SIMD is also important and most applications are not using it. It seems logical to develop new operating system versions designed to take full advantage of really modern hardware. They would add real value to your notebooks, desktop, workstations and servers.

Click here for part 3 where I compare the new capabilities of programming languages and development tools with respect to multi-core processors.

About the author: Gaston Hillar has more than 15 years of experience in IT consulting, IT product development, IT management, embedded systems and computer electronics. He is actively researching about parallel programming, multiprocessor and multicore since 1997. He is the author of more than 40 books about computer science and electronics.


Gaston is currently focused on tackling the multicore revolution, researching about new technologies and working as an independent IT consultant, and a freelance author. He contributes with Dr Dobb’s Parallel Programming Portal, http://www.go-parallel and is a guest blogger at Intel Software Network http://software.intel.com.


Gaston holds a Bachelor degree in Computer Science and an MBA.You can find him in http://csharpmulticore.blogspot.com and http://software.intel.com/en-us/profile/417051/