The Inquiry and Software Testing’s ‘Dirty Little Secrets’

31 Oct

As a former software tester – inquiry into the ‘’ failures seems the perfect forum by which the public could be brought to speed regarding their assumptions that large institutions properly test their customer data and calculated results.  Of course, in reality, these complexities will be lost in the politics (and real concerns) surrounding the ‘Affordable Care Act’ itself, but I can dream, can’t I?

I’ve worked for a major life insurance company, a ‘dot com’, a well known payroll processing company, and one of the original ‘outsourcing’ companies handling ‘pension benefit’ data for client companies.  Though not claiming any specific knowledge surrounding the design, construction, or performance of the healthcare site, I suspect that the dynamics involved were not unlike my own experiences.

I’ve seen major differences in project quality, and accountability, depending on whether software was programmed and controlled entirely ‘in house’ or promised as an outsourced service.  I have also seen differences regarding expectations of quality when comparing the ‘old days’ (simpler in that ‘dumb terminals’ were used by clerks to communicate with company run mainframes) and the modern environment (in which customers interact directly through an endless variety of devices, operating systems, browsers, and internet connection technologies to databases that may be under the jurisdiction of more than one organization).

One of the ‘dirty little secrets’ in software creation is that – for certain segments of a project – ‘automated testing’ was a much more common mainstay in the pre-internet mainframe days.  Ideally, these automated testing tools (themselves ‘software’) are used to quickly and efficiently rerun large numbers of test cases, thus avoiding the need for a tester to sit down at a computer and perform these tasks manually.  In fact, following even a small change to programming code, no claim of quality can be made until all the old (unrelated) processing has been checked for inadvertent changes.  Unfortunately, the likelihood that these ‘regression’ tests have been automated is much lower these days due to the additional complexities mentioned above.

In the old environment, my automated tests were quite easy to maintain.  I simply identified the row and column where data was to be entered, listed the data, and followed this with a symbol representing the pushing of the ‘Enter’ key.  Currently, however, newer automation products need to take into account all sorts of additional environmental factors.  For example – Does the user have other applications open on the ‘desktop’ in addition to the one being tested?  What unexpected (operating system) messages might pop up and how will the testing tool deal with them?  And so on.  Any number of these surprise encounters – not under the control of the actual software being tested – could serve to derail the running of the automated test.  When they do, it is often unclear at first whether the test stopped running because of an error in the software being tested or the testing tool itself!  To make things worse, in order to differentiate their product, companies selling these tools fall into the temptation of ‘feature creep’.  Often, the time spent learning new features defeats its original purpose as a time saving tool.  Unfortunately purchasing decisions here are made by mangers who may not understand the nuts and bolts of the testing environment and/or what makes for efficient testing.  But wait, there’s more!

These automated tests need to run on the actual computing devices they intend to test.  This means that in order to simultaneously test 50 different hardware/software configurations – PC, MAC, different operating systems, different versions of operating systems, browsers etc. – you need to actually have 50 different physical machines available and running at one time!  This, most assuredly, never happens in the actual development environment.  Acknowledgement of the limited nature of today’s software testing is visible as software companies make available free ‘beta’ versions of their latest software in the hopes that users will function as testers by reporting ‘bugs’ ahead of the main ‘release’.  Of course, the availability of this testing strategy should depend on the nature of the software being tested.  You wouldn’t want to hear your bank say ‘Try our latest online banking software and let us know if your checking account balances are being displayed correctly’!

In reality, the amount of time required to create and maintain these automated tests is so high that companies often don’t get around to it.  Especially in cases where projects have been outsourced and there is no expectation of revisiting this (new) software in the future, automation may not be used.  Also disconcerting is that even where a collection of test cases has been automated, the speed with which these tests run depends on the same network traffic issues (live) people encounter.  In the end, for documentation and liability purposes, testers are usually required to put down in writing a description of all the scenarios they want tested, but these may simply end up as ‘cue cards’ for future manual testing.  Late specification changes to the software (such as the government adding new legal requirements) will often mean there is literally no time to rerun these ‘regression’ test cases – either manually or through the use of these clunky automation tools!

It’s also important, in the crisis at hand, to realize that company ‘websites’ are usually not synonymous with the product itself.  In the case of ‘Craigslist’ it may be.  For health insurance it certainly isn’t.  Life decisions may depend on whether calculations occurring behind the scenes (premium quotes, or eligibility for example) were performed correctly.  Simply being able to ‘sign up’ or to see your name formatted correctly does not mean you are out of the woods!  On the other hand, problems appearing in this ‘cosmetic’ realm may be strong indicators as to the extent and quality of the testing performed prior to release.

On a case by case basis, the testing of calculations requires a much larger time commitment than testing simple screen flow.  The tester must manually step through the lengthy and complicated business rules (legal or otherwise) in an attempt to verify that calculated results were spit back as expected.  The endless combination of relationships between input data and rules, such as ‘if this is true, that cannot be true’, often means that testers are only able to test a small sample of the possible scenarios.  With the ‘’ website I would be most concerned with errors in this realm – especially in regards to a law that no one seems to fully understand!

Among my own customer experiences I’ve been victim (or beneficiary?) of a $600 computational error within my auto insurance bill.  This was apparently never noticed, one way or the other, by my insurer.  I received neither ‘Sorry, we previously overcharged you by $600’ nor ‘Sorry, we recently billed you $600 instead of $1200’. In another example, Bank of America – after acquiring the company that previously owned my credit card account – failed to properly handle the cardholder rules as contracted.  Though I had not made a late payment (a prerequisite for any change to my ‘fixed’ rate) they improperly changed my rate to ‘variable’, nearly doubling my interest payment.  I have also heard relayed accounts of inadequate testing performed prior to other large institution ‘takeovers’.  Consumers should really look out in these situations!

Of course, in addition to the issues discussed above, there are also your garden variety concerns relating to things like ‘project management’ methods and outsourcing pitfalls.  A few of these include:

#1) Current trends to be ‘flexible’ and to ‘cross train’ can wreak havoc in terms of learning curves and accountability.  ‘Bug Tracking’ must be handled correctly so that all outstanding issues are individually identified and assigned unambiguously to one, and only one employee at any given moment in time.  Ideally, software testers should be given the power and responsibility of literally ‘signing out’ the end product.  This means that they put their signature down on paper and include notes as to the existence of any outstanding issues (to be fixed in the future).  This process is essentially defeated and meaningless when others, such as project managers are allowed to do this ‘signing out’ in place of those who did the actual testing.  In reality, only my first employer followed this most basic of practices.  Perhaps it creates fear these days in which deadlines and completion dates seem to take precedence over all else.

#2) Outsourcing providers may lack adequate understanding of the client company’s business requirements or existing source code prior to preparing their estimations of the time required for a project.

#3) Employees within the hiring company (like the government) may not understand the technical complications created by late requests for changes to the software.  The outsourcing company is unlikely to tell the client ‘That can’t be done properly within the current time frame’.

#4) In situations where cash penalties might be imposed for late project completion – and especially when coders and testers sense the unlikelihood of any future direct accountability – these employees often avoid being the ‘squeaky wheel’ in status meetings run by the very stressed, less closely involved, project managers.  I saw this become a group behavior on one of my projects in which everyone knew there were major problems, but only I spoke up.  The fear is that the project manager might assume that the person encountering issues is less competent than the others.  In my case, I had documented so many unique concerns along the way – all of which eventually rang true – that no one continued giving me a hard time.  Most employees in my situation would not have risked these waters.

I hope these notes on the true nature of today’s software testing environment prove useful to others diving into this topic.

%d bloggers like this: