I would pinpoint how automated defect identification in documents would have allowed program managers not only identify, but reign in under-scoped & creeping projects & put in place effective project controls. The net effect was to avert out of control scenarios, avoiding programs being placed in jeopardy. Interestingly, two years later this remains our core proposition.

One of the examples I tended to cite was the US Census Bureau with its FDCA (Field Data Collection Automation) program aimed towards automating household data collection for the 2010 US census. The program was initiated in 2006 and by 2008 was in deep trouble. FDCA was all about equipping agents with handhelds that would automate the collection of data at the doorstep, cutting out the big expense associated with manual collection and the subsequent analysis of the data.

The quote on my slide deck came from the US Census Bureau’s Steve H. Murdock, when testifying before the Subcommittee on Commerce, Justice, Science, and Related Agencies Committee on Appropriations, U.S. House of Representatives in April 2008, some two years after the contract for implementation was originally put out for tender and awarded in 2006.

“From the beginning, we did not effectively convey to the contractor the complexity of census operations, and the detailed requirements that needed to be fulfilled in order to complete the operations that FDCA covers.  Once these detailed requirements were completely delineated, we had serious concerns about rising costs and our ability to complete a successful 2010 Census if we continued developing the FDCA program as planned.”

At the time, I remember digging into the details of his testimony along with some general research into the program itself. It became apparent that this specific program failure accounted for between $2 to $3 billion of spend.

So, recently I was travelling on the subway in New York and spotted an ad encouraging participation in the census. It brought back memories of my research and it prompted me to wonder how the census was going from a back office or IT program perspective. Did they ever automate any of it I wondered?

I went to the census website to discover the following little statement at the bottom left of the ‘How it works’ section on the 2010 home page: ‘Can I fill out my form online?’ The answer being ‘No, not at this time. We are experimenting with internet response for the future.’

One wonders if back in 2006, had prioritization been given to offering a secure online capability and more importantly, if program management had leveraged an effective PMO & review process with the vendor, could this failure have been averted?

My guess is that; had the collection of program documents for each project including project charter, BRD, vision etc. been analyzed by VisibleThread, very early warning signals would have been raised. This relates particularly to automated ‘discovery’ and ‘concept tracking’.

Now whether anybody at executive level would have paid attention is a whole other matter entirely.

At least, I can say that our current crop of Fortune 500 customers, mostly in the Financial Services sector, are seeing the merit of early warning signals for their programs and projects. They are equipped with the information that keeps programs on track scope wise. ‘Preventative medicine’ can often be a hard sell at executive level, but enlighted and battle weary stakeholders see the merit. Maybe with a few more $2 billion fiascos, we might make headway as an industry. We can but live in hope.

Until next time, Fergal.

PS: For more context on the whole affair check out: http://spectrum.ieee.org/riskfactor/computing/it/census_going_back_to_paper_due

I recently came across a nice analogy on the PMI site at: www.pmi.org/Pages/PMO-Growing-Pains.aspx comparing a PMO with the human lifecycle. The basic assertion is that PMOs are born, grow up and hopefully end up as mature, ‘adult’ PMOs.

Applying the analogy to the PMO highlights the importance of ‘good parenting’ in terms of how a PMO is set up (i.e. born) and how it is nurtured and evolves. In fact, many PMOs stay at infant stage, not progressing much beyond the basic idea of a group of project managers with no particular strategic imperitive.

Effective PMOs serve to chaperone the program to success, avoiding the inevitable speedbumps along the way to program delivery.

Is this your PMO?

So, if a PMO struggles to mature, can we blame the PMO itself or is it lack of supports in the broader family? In a sense, this is the ‘nature versus nurture’ argument. This is a bigger question that touches on the degree of executive level support for the PMO along with the PMO leadership itself. The better the support structures, the more likely you will have a mature PMO. Of course talented PMO leadership is the second clear contributory factor to PMO success.

So, what does a mature PMO look like and do you belong to one?

Well, in our experience, aside from operation aspects of vendor management etc. mature IT oriented PMOs fulfill three fundamental objectives:

A.) they contribute to successful project management in measuable ways

B.) they ensure projects align with strategic goals articulated at program level

C.) they set effective and appropriate standards for processes and methodologies within the program

To achieve these objectives arriving at the a good level of maturity requires quantifiable & systematic metrics. Here is the rub! PMOs can easily sink in deep documentation and unduely onerous processes attempting to achieve maturity.

Since projects and program goals are typically expressed in terms of vision docs and BRDs, there are pretty simple concrete steps that can tangibly help a PMO out of the infant stage to a more mature level avoiding the documentation overload problem. The answer: more consistent project documentation supported by automation using solutions like VisibleThread. Automated vetting of documents for poor quality is helping drive the PMO from ‘infant’ to ‘mature adult’ with a lightweight approach.

So, I’ve come up for air in the last few days and I wanted to take the opportunity to share some thoughts relating to medium to large programs in IT and some challenges I’ve being seeing at first hand.

Many of the people we work with are involved in programs typically coordinating between 5 to 15 projects. These roles are variously referred to as ‘program managers’ or sometimes ‘program PMOs’ (Program/Project Management Offices). They look to VisibleThread to help assess the quality of documents for these initiatives.

For instance, one of the large programs comprises 15 separate projects spanning 3 separate locations globally. In this case, we have 15 vision statements, 15 BRDs (Business Requirements Definitions) and will in due course have multiples of 15 for other types of documents including FRDs, Arch specs, Test Plans etc.

This program may well ultimately yield 15 x 5 (1 for each type) documents, i.e. 75 separate docs. If we assume that each doc averages 24 pages, we’re looking at 1800 pages of content. This is, believe it or not, roughly the size of the King James bible! (well at least this edition: http://www.christianbook.com/21st-century-king-james-bible-hardcover/9780963051233/pd/53495 )

Consider this from an individual author’s perspective. A project manager authoring her own project BRD (assuming it’s one of the 15) will likely be conscious of the considerations involved in that specific project but less conscious of sibling document sets for adjacent projects that are part of the program.

The challenge of assessing risk within the content is obvious; different authors, different audiences, different levels of detail, potentially large volumes of content making gap analysis and interdependency analysis very difficult indeed.

Working with people on the ground, this has been one of the biggest issues at the overall PMO or Program Management role level. One customer mentioned an example where the far east team had a definition of ‘client’ that was subtly different from the US version. Unravelling this interdependency did not occur until into the UAT phase and the consequent fallout was material in the context of the overall program, resulting in cost due to rework and a significant delay in the delivery timeline.

Dependencies typically go in two directions, horizontally across sibling documents (eg: Proj ‘A’ BRD relating to Proj ‘B’ BRD) and vertically, down to different document types (eg: Project ‘A’ Vision doc relating to Project ‘A’ FRD) and different sections in these docs. These are clearly non-trivial challenges and very difficult in the context of medium to large program initiatives.

Can’t we use a Traceability Matrix?

Many people within the business analysis community tend to suggest using  a Trace Matrix as one way to tackle these issues. A fairly simple image of such a matrix (taken from wikipedia) is shown to the right. Effectively a trace matrix seeks to identify the connections between numbered requirements.

In my experience, trying to use a trace matrix for the above issue is near to impossible. Three factors make this so: 1.) the effort to create and maintain an up to date trace matrix in light of changing underlying document content is unduly onerous, 2.) a trace matrix is not scalable visually beyond the simplest of initiatives, certainly not with 1800 pages of content and finally and perhaps most importantly 3.) expecting senior stakeholders such as program managers to go to the effort of maintaining such a trace matrix is ‘a bridge too far’ in most organisations and will simply not happen.

So, enter the ‘discovery’ view within VisibleThread. Discovery takes a different approach. Rather than forcing an explicit marking of ‘traceable’ relationships, it instead automatically scans a body of documents and calculates the occurrence of ‘nouns’ in the context of where they occur within document content.

How does this work?

The VT Server uses NLP (natural language processing) to automatically scan raw documents and store occurrences. The dashboard then displays these occurrences cross referencing the document content.

Let’s say we are looking at documents concerning an extension to an existing trading system that will affect business rules around how accounts are validated for trading thresholds. In this case you would expect to see a reasonable distribution of certain key domain concepts like ‘trade’, ‘dealer’, ‘letter of credit’, ‘account’ etc. spread across certain documents in certain distribution frequencies. In particular I would expect to see quite a bit of content and rules relating to ‘account’. The BRD, FRD and use cases that touch account manipulation eg: ‘Update Acount’ and associated test cases would all be expected to have relatively high occurrences of the concept ‘account’.

Looking at this screenshot, we see the list of ‘discovered’ nouns.  The grid in the center shows the correlation between concepts (nouns) like ‘account’ and in what documents they occur. Each document is represented by a column adjacent to the checkbox. Here we see 5 columns representing; a BRD doc, a tech spec doc and 3 use case docs.

In our case, only 2 occurrences of ‘account’ are encountered indicated by a green dot in the specific document column. Those occurrences are *only* in the BRD doc meaning we have no reference to ‘account’ in the detailed tech spec doc or in any of the use case docs, a potentially serious gap. We can spot gaps/issues without having to trudge through the entire set of documents in this way. For program managers at high levels, this ‘surgical’ view focussing on inherently important concepts top down, means risky gaps and defects can be identified and eliminated.

Were these types of gaps to remain undiscovered in our ‘bible’, (as they were in the above ‘client’ example) we would be in for a rude awakening downstream. Thankfully, with VisibleThread we are seeing how we can avoid these issues in very practical and intuitive ways, marrying NLP with strong visualisation techniques.

all the best,

Fergal.

PS: Over the years I have encountered some insightful people attempting to use mind maps to understand dependencies relationships. It is certainly a good idea on the surface but just like the trace matrix challenge, the mind map itself (albeit electronic in nature) must be separately maintained outside of the content. Just like trace matrices, this is hard to do when the body of content becomes in any way large.

So, it was extremely refreshing to find a great article just published this week over at www.ModernAnalyst.com titled: ’ Using Extreme Inspections to Significantly Improve Requirements Practice’. The article focusses on applying Quality Assurance techniques to IT documentation. It’s written by German engineer Rolf Goetz, who has obviously been at the coal face and clearly appreciates many of the challenges we all face in establishing more empirical ways to improve process.

Much of what Rolf talks about leverages work from Tom and Kai Gilb and others.

What I found refreshing, is that the article shines a light on the area of quality, similar to our mantra at VisibleThread. As ModernAnalyst (where this article was published) is a relativly mainstream portal for BAs, highlighting the value of inspections to this much wider audiance is great to see.

The emphasis in the article is on collaborative inspections. Rolf and indeed many of the proponents of these techniques, prefer the term ‘inspection’ over ‘reviews’, specifically suggesting ‘inspection’ of a random set of document pages yielding clear metrics around defects that can be extrapolated to the rest of the doucment(s).

I’m not so sure I care as much about the nuances of terms like ‘inspection’ vs ‘review’. I do however very much agree with the notion of spot checking a random sampling of document pages frequently rather than taking a more big bang ‘gated review’ approach. The latter is by definition somewhat more reactive and less effective that the former. Either approach however is preferable to the status quo in many organisations that we come across, i.e. little or no formal review.

So, whether we have frequent inspections or more back ended review of specifications, any kind of early intervention during the analysis phase and formal/informal review or inspection process is to be heartily welcomed. Too often, many organisations do not have even rudimentary vetting/validation procedures in place.

What was doubly exciting for me in this article is that you see the actual ROI of inspection in clear terms. Utilising the approach of ‘extreme inspection’ in one case Rolf cites, we see a reduction in the number of defects per page by 50%. Clear empirical evidence of actual defect reduction as a consequence of inspections in real projects is hard to come by and so Rolf’s case studies are useful to consider.

Broadening out the discussion; we’re currently working with a number of major customers especially in the financial services sector & without exception, the key goal is to establish an effective review (inspection) process backed by automation and very clear metrics.

In this respect, Rolf’s article is timely and very welcome in that it shows clear evidence in a ‘real-world’ example of what is possible from a defect detection perspective in specifications, albeit in a manual way. We are seeing that automation via VisibleThread can substantially magnify the efficiency of the rate of discovery of these defects, indeed often allowing inspection where heretofore resourse pressures have made it difficult to pull off.

I’ll be interested to see more of this kind of of coverage in the main stream media. My hope also is that VisibleThread can allow wider exposure and adoption in a meaningful way of the types of techniques described. Now there’s a very exciting thought…

If you’re interested in looking into this area more, Tom Gilbs work can be found at: http://www.result-planning.com/Requirements and Niels Malotaux at http://malotaux.nl/nrm/pdf/ReviewInspCourse.pdf

Fergal.

“I work for a small subsiduary. Can you recommend sources of documentation templates so I can build best practice to model for our parent company? Then I could recommend visible thread to propagate the standards.”

The question opens up the very interesting, highly subjective & contentious topic of software process & the ‘best’ document template. I thought I’d blog on this and share an extract of my answer as it may be of some help to folks considering this whole area. Not a simple question, but here was my answer:

— A number of questions have a bearing on what type of documentation you use in the lifecycle. I like to consider these things:

• Size, duration and distributed nature of projects? Larger projects require more controls, distributed projects also require more controls. Long duration projects also require more controls. (i.e. gates in your process and associated templates)
• Size of projects, are these projects you work on mission critical or not, or a mix? The former require heavy oversight, whilst they can be ‘agile’ in very sophisticated organisations, they typically are waterfall if you look closely. Less mission critical projects require way less docs and can be far more flexible in the depth of content captured.
• Style of projects, are these projects you work on waterfall in nature or iterative (or agile)? In reality, many organisations claiming to be iterative/agile tend to be ‘fakes’ when you dig to any extent under the covers, in other words whilst they may have that tag they will turn out to be waterfall. Either way, both approaches require change and need to be measured for change over time in terms of document content
• Culture of your organisation, is your org open to adopting better practices, are they able to absorb change quickly? If your org culture is somewhat resistant, I would recommend getting some kind of buy in from management & gradually introduce revised templates over a period of say 2-4 months, addressing certain key areas initially, eg: get a decent func reqs spec in place along with an associated non-funcs spec template in place is always an excellent start.

Regarding templates here a few sources to get you going:

1.) A requirements template I always recommend is James and Suzanne Robertson’s Volere, which can be found at: http://www.volere.co.uk/template.htm This used to be a free download, although I see James and Suzanne are now charging a nominal fee. Worth it however as it is very strong and covers out a wide set of considerations in requirements analysis.

2.) Another good option is to search docstoc (or scribd) for templates. Here’s one that I like (and this ships as a sample set in VisibleThread) for non-functionals: http://www.docstoc.com/docs/287018/Non-Functional-Requirements-Document-Template

3.) There is an open source project at: http://readyset.tigris.org/ It attempts to compile and assemble a fairly complete collection of templates for the various states of the software lifecycle. Specifically the full list of templates is at: http://readyset.tigris.org/nonav/templates/frameset.html Whilst it is an interesting project it comes out of academia and is very light on the early stages of articulating requirements but better on later stage docs. Worth a look but I would be inclined to cherry pick and augment with better sources.

We tend to base our guidance on using VisibleThread as the way to help power your approach. VisibleThread does come with a number of very good Best Practices out of the box (based on our own experience and on service partners practices) but in fact any Best Practice template can be used and applied in VisibleThread within 5 minutes to form the basis of any quality and process improvement initiative.

I was travelling with a colleague in New Jersey and we were running slightly late for an appointment with a customer, feeling a little anxious as the clock ticked. We stopped to fill up on gas as the yellow ‘low’ gas indicator light appeared on the dash. I instinctively made to get out of the car and start filling up.

Well, you can imagine my surprise when my colleague gently restrained me, explaining that it’s against New Jersey state law to pump your own gas.

To quote a certain Mr. McEnroe ‘you cannot be serious’. The incident was borderline comical, or at least to me at any rate, as we idled in the car awaiting the visitation of the gas attendant before we could proceed.

The incident prompted me to reflect on some of the more interesting aspects of what we do in software. We tend to apply the same laws and process to projects universally, often putting a block on proceedings for no other reason than that ‘the process says so’.

So, in many companies, differing styles of projects can often have identical process gates and approaches, where there is a patent difference in risk factors. A 3 month, 2 person effort for an internal audience is a dramatically different proposition than a multi-year mission critical project. Yet in many cases, the Best Practice for each style of project is not considered, nor indeed does common sense intervene, in fact in many cases the same process gates and phases apply to both. This either makes the small project exceedingly bureaucratic or conversely allows the larger project to be too loose.

So, this brings me to VisibleThread. One of the interesting benefits we see as we work with corporate users is the products ability to tailor specific Structural Best Practices for the different types of documents. For instance you frequently see a specific BP (Best Practice) for ‘Business Justification’, a BP for Functional Requirements, one for Non-functional etc.

What has become more interesting however in some recent examples is the ability to slice along style of project. As James and Suzanne Robertson (http://www.volere.co.uk/index.htm) refer to them; the rabbit projects (small), the horse projects (bigger, more mission critial) and the elephant projects (decades of engineering years, expected lifetime of the live system approaching a decade or more). So, in essense we begin to see structure Best Practices for:

• ‘Rabbit - Functional Template’
• ‘Horse - Functional Template’
• ‘Elephant - Functional Template’ etc.

Now, I guess if we could only get a Best Practice for ‘Person in New Jersey, late for a customer appointment, who needs to work the self-serve gas facility’ we’ll truly be in business

Background: If you want to find out a little more about the background on this ‘interesting’ law, check out: http://ask.yahoo.com/20040715.html As it turns out, this law was put in place in 1949. The purpose of it was to protect consumers and gas station owners from costly, and possibly deadly, accidents. All perfectly sensible when you think about it, back in 1949! But 60 years later, it’s an anachronism.

It was in this context that the particular pilot customer brought up the term ‘Center Of Excellence’ or ‘COE’. They were kind enough to share the definition that they have come up with.

For them, it involves ownership of the Business Analysis practice area to support effective elicitation and management of requirements and related artifacts as well as putting in place the tooling and disciplines surrounding. The particular customer feels that VisibleThread can play a central role in the oversight and support for their COE, which is nice because it certainly is a sweet spot of VT. What’s also interesting is that the COE has the buy-in of the executive management team.

But wait, wasn’t this really a PMO (Project Management Office) in disguise I thought to myself? Well actually not quite, in fact distant cousins as it turns out.

PMO’s tend to be concerned with effective project management practice, focussing on the entire lifecycle, all aspects, ranging from resourse management right down to effective delivery practice. I guess the key difference is that the COE (or RCOE, Requirements COE) concept is really very focussed on Business Analysis and sponsoring Best Practice in that area. A second clear differentiator from the classic PMO is that the COE is also actively involved with ‘in-flight’ projects, playing an active mentoring role. This marks it out as quite a different animal to a PMO, at least in my experience. This latter hands-on aspect is one of the most important enablers for successful process improvement in my opinion, as well as of course executive sponsorship.

The conversation did prompt me to dig into the whole area of Centers of Excellence. After a quick survey of some of our sales guys (and specifically what they’re hearing in the field), it turns out COEs are popping up quite a bit and have become fairly common in the IT space over the last year or so. If you’re interested in looking at this further, here is a succinct & useful outline of the COE concept from the IIBA website: http://download.theiiba.org/files/resources/CoRE.pdf

Now, it’s not like this is completely new. The RCOE concept has been around for some years, albeit in many cases called something else. For instance, LogicaCMG, in the Netherlands, whom I did some work with about 5 years ago had a very strong ‘Competency Office’ in place. This effectively conducted the same type of activity as the COE. What is refreshing however is to see the traction behind this in the US. If it only serves to prompt a discussion at executive IT level about the importance of high quality documentation (with clear metrics), it will have served a very good purpose.

Now I have to say, I’ve never met Don, but for anyone who cares a whit about the user experience and good design, this book is a must. The reason I bring this up is that I am always surprised how little appreciation there appears to be for usability in the area of software products.

I don’t mean graphic design, I mean getting right inside the head of your user or intended user, understanding what flows and work practices will be most meaningful in a product context and when, often out of ignorance for a certain community of users, vendors design something that will basically alienate the user base.

And so, this brings me neatly to the latest in a long line of offenders in this category… deep breadth, gmail.

Now, I know there are a gizillion people using gmail, among them a lot of very zealous advocates. Plus, it’s not trendy to suggest there could be anything amiss with gmail, in fact it’s gotten so pervasive that the excellent yahoo mail interface has taken on noticeable gmail-esque hues, right down to those oh so square buttons. But… believe me, for a corporate user to not be able to: sort on the ‘sent’ person, categorize mail into folders (not tags), and most annoyingly of all, be forced to consider mail threads as discussions with no easy way to find specific mails within a thread sent by a specific person on a specific date, it makes serious adoption of gmail nearly impossible in a corporate context.

In fact, I was chatting with one of our sales guys and he tells me he has taken to changing the message title when replying in order to split the mail threads when using gmail.

Absolutely gmail has tons of advantages, SaaS being the biggie, aside from the obvious cost/or lack thereof, but as long as it has an interface that flies in the face of accepted norms & genuinely reduces productively for corporate mail users it will never penetrate the corporate environment.

Now, if you have never felt this pain, indeed many people I talk to haven’t then carry on using gmail, but as long as a huge body of potential users are alienated then Outlook will continue to live a long and happy existence. Trust me, I’m no great fan of outlook but after a full year of gmail as my primary interface, I’m finally back to it as of a couple of weeks ago. Ah, the bliss of actually being able to find mails by sorting on sender and being able to categorize them into folders again… I don’t know myself.

Well, anyone who has done this kind of thing will tell you a.) it’s easy to underestimate the effort in terms of time and b.) there really isn’t a very effective instruction manual, with plenty of trial and error involved. Not a great surprise then, it took about 4 months to build, lots of evenings/weekends hauling blocks and assorted lumber of one sort or another.

Shed Version 2 evolving

Rolling the clock forward, last year we had moved to a new house. What did we absolutely need? yes, that’s right, a shed. So this time, somewhat older & wiser and with some good experience under my belt, I set out to craft my masterpiece, shed version 2. This time I solicited some help on laying the foundations realizing that getting them level is pretty important, did a much better job on the roof joists and generally did a far better job in about 2/3rds the time.

So, where am I going? Well, co-incidentally, around about the same time that I was building shed number 1, circa 8 years ago, I was also embarking on my first startup; a company in the Requirements Management space called SteelTrace. Indeed, if you’re interested you’ll see this particular ‘shed’ still alive and kicking, it now lives over at Microfocus (http://www.microfocus.com/products/OptimalTrace/Enterprise.asp), albeit a few owners later via my friends at Compuware (http://www.compuware.com)

… and last year, around about the same time I started shed version 2, was also when I started seriously reflecting on the state of where we are at in IT, in particular on why project failure rates are still so crazily high (http://www.standishgroup.com/newsroom/chaos_2009.php). Despite my best attempts to make things better with SteelTrace, Standish were telling me things hadn’t got that much better at all. In fact, if you believe the stats, they’d got a whole lot worse!

As with shed number 2, I realised there may be better ways to do things, project success may not be assured after all, simply by adopting a Requirements Management tool.

So we come to VisibleThread. VisibleThread is about first principals. Going back to the basic premise, if people are doing stuff in MS Office, let’s see how we can make the IT process better, yielding better outcomes for the business. Rather than ditching the tooling that 90% of people use, let’s see can we get better quality and better outcomes by embracing MS Office and offering better insight inside the documents, making them come alive. In a nutshell, that’s what we’re up to here at VisibleThread.

In this blog, I will try to share  some (hopefully) interesting observations, giving you an insight into what myself and the VisibleThread team are doing to achieve this goal.

I hope you’ll join me in this new journey. If we have half the craic (as we say in Ireland) along the way, make new friends and solve some real thorny issues while doing it, that will be a great outcome. If you want to stay in touch, why not subscribe to our RSS feed or join us over at linkedin in the ‘Friends of VisibleThread’ group.

hope you can join me, Fergal.