Sunday, 25 March 2012

Monday morning blues

On Monday morning, I was on my way to work when I was called on the cell phone asking if I knew that the Production system was running slow. I didn't, but said I would look at it as soon as I got in. While I was getting set-up I had a couple more calls from people to tell me that things were not going well.

When I eventually logged on, I saw that there was a job that had been running since the Friday before - and it was logged as being one of the consultants that hadn't been on site for some time. I asked around and eventually was told that this particular consultant was due to be on site - he had been organised to look at a particular issue.

When they guy actually arrived, he looked thru the system and checked out what was going on. It turns out that he had started this job remotely because he needed to check on some data - he thought it was going to run for maybe 6 - 8 hours. Instead it was still running after 60 hours and only finished later that day. It also didn't do what he wanted.

After he did some further checks, it turns out that a change had been made to the system that would actually cause an issue with the program that he had tried to run. No-one knew about the details as the guy that made the change is long gone, out of the country and there were no references to the work anywhere. In fact the consultant onsite wasn't entirely sure what change had been made. He did say that it had only been done in the production system - there was no transport for the change, and when he tried his program in the Dev system, it ran OK.

It now appears that we are going to have to pay to get one of their guys in to look at this now - they need to try to figure out what was done and then first correct it, and second, try to do whatever job it was supposed to do properly. I asked if the work would be  FoC (well it was their guy's mistake) but no such luck. In fact, they don't actually know how long it will take to investigate the issue - possibly 4 -5 days which we will have to pay for before they tell us how much it will cost to fix.

This highlights a couple of points. First of all, most of the work that the SI has done has never really been documented, even tho' that was a point raised right back at the beginning. We were told categorically, that they would document all changes that were made - in fact, we have very little evidence for any of their changes.

Another major issue, is that a lot of the work has never really passed any form of Quality Control gateway test. Again this is an issue that was raised on many occasions, and we were assured that we would have the power of sign off on everything. Yes there have been items that we have signed off on, but there are a ton more that we haven't. In some cases, we don't know that work has been done, and are only finding out when things go seriously wrong, and investigation uncovers the root cause.

I've spoken to numerous people and the documentation that we have been given by the SI right thru the business is so poor as to be virtually useless. We have very little information on a lot of the changes, and it's difficult to see what we can do now other than get a specialist in that can uncover some of these, and hope that we can get them documented in future.

I've also highlighted that they often ignore good practice - and here was another example. A change was made directly in the production system without having been tested in the other systems. I'm also concerned that some of our people have learnt these bad habits - I regularly get asked to do something directly in PRD, and if I refuse and point out the correct procedure, I get an argument that the consultants do it that way.

We should have had a proper change management procedure enforced from the beginning, and I will say that when I queried this, I was told that SAP had a robust procedure that the SI staff would follow. There is a procedure, but the consultants don't bother to use it, and it sometimes seems that they see it as a PITA to be avoided rather than good practice to ensure that we don't get problems.

For me tho' the big issue is that all of these problems are completely avoidable - there is simply no excuse. I wish that I had a way to identify just how much of the money that we have paid has actually be wasted on work that does not do what it should, or has never been tested correctly.

I just hope that we don't get off to another poor week this Monday.

7 comments:

  1. I've been following this blog for a while now, on occassions. It's fairly clear by now that you are being had (sic) by an SI and for some reason keep up with that - is it the small print?

    You can raise issues all you want, but unless you slap some punishment and reward system on top of that, accompanied by dates, times, signatures and other properly measurable, verifiable, and above all traceable means and methods, it's just water under the bridge

    I am looking forward to a post in which your company decides upon clear and candid measurements to control this wild-west story

    Here's a few examples that might help, quoting from this post:

    "We were told categorically, that they would document all changes that were made"
    - Fine, but how would they document what, according to which templates or examples, within which timeframe, store it in which DMS or CVS, etcetera?

    "we were assured that we would have the power of sign off on everything"
    - How nice. When, at what stage, what value would that add, what power would that wield, what would be the significance of it, and the consequence(s) of a non-sign-off?

    "the documentation that we have been given by the SI right thru the business is so poor as to be virtually useless"
    - Of course it is, but you don't have any tangible agreement with them on the quality, let alone even the quantity of it, do you?

    "I get an argument that the consultants do it that way"
    - And what do you do then? Follow suit? Yup

    Let me point you to a very wise rewmark you make yourself:

    "For me tho' the big issue is that all of these problems are completely avoidable"
    - yes, contracts and agreements usually do that, written down and mutually accepted

    I can tell you right now that you will be in the current position months, if not years from now, unless you take decisive action RIGHT NOW.
    Yes, you have wasted a lot of time, energy and money in the past, but how about the future? How long do you think that you're going to be knee-deep into this mess? What if? This might even bankrupt your business

    So, please: next post should be a post in which you tell or ask how to go forward with this - I am sure dozens of people can give you proper advice for free

    ReplyDelete
  2. 1) Lock the SAP production down using tx SCC4.

    2) enable logging for the table T001 in SE13

    3) Remove ALL logons from you consultants and FORCE them to do work though your local people.

    4) If you know who did the change you can search for transports in SE03.

    5) Advertise for a contractor to work for you and reporting to you no the SI. And bring some resource in house. Will be far cheaper and better than using a SI.

    ReplyDelete
  3. Also use tx SM50/SM66. These will give you an oversight of ALL processes running in your system. SM50 allows to cancel an batch jobs that are causing issues.

    One job shouldn't cause a well specified SAP system to fall over. If you want to see who is in your system use AL08.

    Also remember that ANY changes to any SAP standard programs are logged under tx SE38 - Utilities - Version - Version management.

    So if a program crashes with a ST22 short dump. - Check the transaction, you can find our who has made changes to that program.

    Every change in SAP is logged by SAP. You just need to run the right reports.......

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. You have done a SPECTACULAR job of maintaining a journal of the FRAUD that has been committed. Why don't you approach your management about suing the SI? I'm so tired of hearing about the fraud so many companies experience. And that is why I have put together these posts with specific instructions and details to ensure you do NOT get ripped off by some of these con artists!

    ==================

    I have no issue with H1B’s, student visas, etc., but these folks should be willing to work their way up just like many hard working folks rather than wreck your business and damage the industry through fraud. For more extensive insight into the problem, AND specific methods for dealing with it, please see some of the other posts:

    -- ERP Consultants: Is the Promise of Knowledge Transfer just part of the Sales Pitch?
    http://www.r3now.com/erp-consultants-is-the-promise-of-knowledge-transfer-just-part-of-the-sales-pitch

    -- Protecting Yourself from SAP Consulting Fraud (SIMPLE method to screen out the SAP FRAUDS!)
    http://www.r3now.com/protecting-yourself-from-sap-consulting-fraud

    -- Screening and Interview Methods to Find the Right SAP Consultant
    http://www.r3now.com/screening-and-interview-methods-to-find-the-right-sap-consultant

    -- Screening and Interview Methods to Find the Right Consultant – Part 2
    http://www.r3now.com/screening-and-interview-methods-to-find-the-right-consultant-part-2

    -- A Cautionary Tale About SAP Knowledge Transfer
    http://www.r3now.com/a-cautionary-tale-about-sap-knowledge-transfer

    -- SAP Technicians or Experts
    http://www.r3now.com/sap-technicians-or-experts

    -- Successful SAP Project Team Composition – Technicians or Experts?
    http://www.r3now.com/successful-sap-project-team-composition-technicians-or-experts

    ReplyDelete
  6. Martin, James, Bill,

    Of course you are absolutely correct in what you say. However, you seem to assume that I have some authority in the project - I can tell you that I have had none whatsoever for some considerable time. In fact, I have been kept out of project meetings over the last 8 months. I only get involved now when something has gone wrong - and usually after someone else has already tried to fix the problem and made it worse.

    I have discussed the more serious issues over good practice with various managers, and it does seem to me that as far as they are concerned, I am available to fix any problems, so there is no issue. (As long as they have a fire extinguisher, they don't need to keep the gas container closed, right?)

    I have raised the situation with senior management and what have they done? Zip - and it now seems highly likely that they will continue do nothing. As for why, I wish I knew.

    Is this frustrating? - you had better believe it. So much so that I have been looking around for other work for some time. I have in fact almost reached the stage where I would rather work behind the counter at McDonalds rather than continue to do what I am currently doing - and I can assure you that is NOT an exaggeration.

    ReplyDelete
  7. Hi this is shiva kumar i am working on sap bwbi.. i just browsing blog s on hana there i found your blog is interesting .. i like to say thank for sharing a information on sap sap-security

    ReplyDelete