Monday morning blues

On Monday morning, I was on my way to work when I was called on the cell phone asking if I knew that the Production system was running slow. I didn't, but said I would look at it as soon as I got in. While I was getting set-up I had a couple more calls from people to tell me that things were not going well.

When I eventually logged on, I saw that there was a job that had been running since the Friday before - and it was logged as being one of the consultants that hadn't been on site for some time. I asked around and eventually was told that this particular consultant was due to be on site - he had been organised to look at a particular issue.

When they guy actually arrived, he looked thru the system and checked out what was going on. It turns out that he had started this job remotely because he needed to check on some data - he thought it was going to run for maybe 6 - 8 hours. Instead it was still running after 60 hours and only finished later that day. It also didn't do what he wanted.

After he did some further checks, it turns out that a change had been made to the system that would actually cause an issue with the program that he had tried to run. No-one knew about the details as the guy that made the change is long gone, out of the country and there were no references to the work anywhere. In fact the consultant onsite wasn't entirely sure what change had been made. He did say that it had only been done in the production system - there was no transport for the change, and when he tried his program in the Dev system, it ran OK.

It now appears that we are going to have to pay to get one of their guys in to look at this now - they need to try to figure out what was done and then first correct it, and second, try to do whatever job it was supposed to do properly. I asked if the work would be  FoC (well it was their guy's mistake) but no such luck. In fact, they don't actually know how long it will take to investigate the issue - possibly 4 -5 days which we will have to pay for before they tell us how much it will cost to fix.

This highlights a couple of points. First of all, most of the work that the SI has done has never really been documented, even tho' that was a point raised right back at the beginning. We were told categorically, that they would document all changes that were made - in fact, we have very little evidence for any of their changes.

Another major issue, is that a lot of the work has never really passed any form of Quality Control gateway test. Again this is an issue that was raised on many occasions, and we were assured that we would have the power of sign off on everything. Yes there have been items that we have signed off on, but there are a ton more that we haven't. In some cases, we don't know that work has been done, and are only finding out when things go seriously wrong, and investigation uncovers the root cause.

I've spoken to numerous people and the documentation that we have been given by the SI right thru the business is so poor as to be virtually useless. We have very little information on a lot of the changes, and it's difficult to see what we can do now other than get a specialist in that can uncover some of these, and hope that we can get them documented in future.

I've also highlighted that they often ignore good practice - and here was another example. A change was made directly in the production system without having been tested in the other systems. I'm also concerned that some of our people have learnt these bad habits - I regularly get asked to do something directly in PRD, and if I refuse and point out the correct procedure, I get an argument that the consultants do it that way.

We should have had a proper change management procedure enforced from the beginning, and I will say that when I queried this, I was told that SAP had a robust procedure that the SI staff would follow. There is a procedure, but the consultants don't bother to use it, and it sometimes seems that they see it as a PITA to be avoided rather than good practice to ensure that we don't get problems.

For me tho' the big issue is that all of these problems are completely avoidable - there is simply no excuse. I wish that I had a way to identify just how much of the money that we have paid has actually be wasted on work that does not do what it should, or has never been tested correctly.

I just hope that we don't get off to another poor week this Monday.