Saturday 31 January 2009

Keep it in the family

I suppose at this point, I'd better make some comments about the wife and kids. They have been really great putting up with all sorts of issues. I sometime get home late and in a foul mood; they have put up with an awful lot and deserve to have that recognised. I'm not the only one -the company held a party before Christmas and the wife and me went along. She got talking to a couple of other wifes (don't they always!) who have also had to put with stressed out husbands.

She told me after that 2 of the other managers are heading for divorce - although they may not know it yet. The wives have had enough of late nights, weekend working, cancelled holidays. I've spoken to one of the guys - he missed his son's first game of the season last year, which was the first time the kid had actually played. He was really angry, and I think that if it continues as it is, he will just quit. In the current economic conditions, that's not a good idea - but he said privately that he would rather take a 50% paycut than continue as they are. (Although I'm not sure he really would).

The family got away on holiday last year without me - I have the use of a condo on the coast for a couple of weeks. My eldest girl thought that was OK as it turned into a girl's holiday with my son away at camp. She made a point of telling me that my wife had gone out dancing with them and had had a great time. I wish I had been there, but they deserve the break.

Basically, the consultants are really arrogant - they arrive and expect everyone else to be immediately available. It doesn't matter that we have a business to run, or that they haven't arrange the meetting; we have to be there. If we ask them to do something tho it's a different matter. They can take as long as they like, and if we dare to question why it takes so long, it becomes a real issue.

One of my guys has been involved in loading data from the other systems. He took a day off to take his little girl to hospital as she had taken a fall and needed some attention (she was OK later, but stayed in the hospital for a couple of days). The consultant project manager actually phond him on his cell phone and demanded that he come back to work as they were trying to resolve an issue with purchasing records. He wasn't happy when I explained to him that if he ever does that again, he will have the cell phone lodged in his lower colon.

A short one today; I'm tired and need to rest up.

Friday 30 January 2009

Security - we don't need no steenkin' security

Shortly after I joined the company, I spent quite a bit of time (including time at home) working on putting together some policy documents. These dealt with a number of issues including security. In fact, I am certificated for ISO 27001 and after about 2 years, I managed to get the company audited successfully. It was hard work but worthwhile - it has surprised more than a few people how useful that accreditation has proven to be. In several cases, our senior sales people were told by customers that the final decison to buy from us came about because we were accredited for ISO 9001 & 27001, and our competitors were not.

When we started on the SAP project, I presented the consultants with the relevant paperwork. This detailed what security standards we expected them to reach, what we would allow and what we would not. This is not designed to stop people from working - but it is to ensure that a good level of security is maintained and that the data and systems are protected. It also included an agreement for them to sign.

They were not happy with this - I received regularly demands to reduce security level to allow them to do things. When one of their guys came in to install the servers, he wanted full admin access and our network security account details. You can guess what he was told - in plain language, take a hike. We were told that they were working to ISO 27001, but it is quite clear that they are not and that they haven't the faintest idea of what the standard is about.

They want a permenent VPN connection to our systems - OK that need not be a problem, but it allows any of their people access - they have a lot of staff (I asked how many, but they wouldn't tell me) any one of whom can connect to our system at any time day or night. We regularly find them looking around the systems - we've set-up internal blocks so they only have access to the SAP servers. After all why would they need access to anything else? We have intrusion detection facilities and it regularly gets tripped by someone trying to find a way into areas that they are not allowed (2-3 times a month).

What does get me pissed is that we've identified that several of their staff are using a single acount, so it is impossible to identify who did what. They create accounts in the SAP system for people and we have no idea who they are or what they do. Worse, our AV has picked up stuff that has come in from them a number of times - a couple of them were really nasty trojans and a couple of keystroke loggers.

And this takes me to another topic - the number of consultants. We were told that we would need 6 people; 4 main workers, the project manager and a specialist for the finance. In fact, I have lost count of the number of different people that been involved - some have been to one or other of our sites, some I know have connected via a VPN connection and I believe that there are about 8 or 9 that have connected using other peoples accounts. As far as I can tell, we have had at least 41 different consultants and probably more.

At the beginning, we were quoted a set number of days consultancy work to do the implementation (370). In fact we are now well over 800 days, there are another 65 booked for the next 10 weeks, plus, it seems clear that we are still no closer to go-live than we were 5/6 months ago and we will need yet more people. In addition, during the work over the last 2 years, they keep insisting that certain things were not part of the "original blueprint" and that these items are part of a "second phase" of work. Well we certainly never agreed to any such "second phase". I was told by the project manager quite confidently that we should expect to budget for at least another 70 - 100 days of consultancy work each year for the next 5 years.

Now the worst bit - like a lot of companies, we have had to do a lot of work to implement Sarbanes-Oxley. Yes it was a real PITA, but we got there on our old systems. The first one of their financial consultants insisted that SAP automatically complied with SOX, and that may be true - but a process he tried to put in place most certainly did not and the FD went ballistic when he found out. He actually kicked the guy out (had security escort him off the premises) and refused to allow him back on site.

The consultant firm replaced him with a woman who really seemed to know her stuff - for a while the FD began to think that maybe it would all come together. But then she announced she was leaving to have a baby and they gave us a new guy - he barely spoke English. After a couple of weeks, the FD had him kicked into touch as well because he just didn't know enough about the finance system - he was then replaced by another guy. This one does seem to know about it, but he is crap at explaining things - he is also less than careful, and we keep finding that he has done stuff on one system, but not the others. As a result, some (a lot) of the testing has been a total waste.

Oh well - another day, another dollar. I'm going for a beer. More later

Thursday 29 January 2009

All your base are belong to us

I've spoken to a number of people about SAP installations - it seems that ours is par for the course, although in a couple of areas, we seem to be doing worse than average.

One of these seems to be because they really don't seem to think that we should be running our own hadware. Their head salesman tried to get us to sign to outsource the systems to their datacenter - not quite sure where it is as he was a bit vague, but I think that it is probably India. He quoted a price and then when the FD stopped laughing, he suggested that the company could get rid of me and the savings would cover the cost of that. In fact to cover the cost, they would have to get rid of half our IT team - and the company are not going to do that.

Since then, they have gone out of their way to avoid giving us information or provide adequate training - generally, we get given these enormous .pdf files with the high lever overview and damn all information on how to do anything. We've also had a couple of PowerPoint presentation files with some more info - but often they are incomplete.

We had a problem after the initial installation - they set-up the server, but failed to configure the SQL server properly. As a result, within a week the system ground to a halt and no-one could use it. Now it wasn't too serious at that stage as we were still not actually doing any work on the product, just watching endless presentations about how wonderful SAP is. However, I did keep on at them about getting the system up and running. In fact it was almost 2 months before the problem was resolved, and that was down to myself and one of my staff as we got hacked off waiting for them to do anything - fortunately, we have some experience with MS-SQL and were able to identify the problem and get it fixed.

A month or so later, their head honcho came to site and tried to blame the original problems on us - when it was pointed out that it was their staff that set it up, he tried to say that they left it fully operational, but that we had messed about with it causing the problem. We aked for them to prove this, but of course they couldn't - but that didn't stop them from trying to pin the responsibility on me and my guys.

In fact, that seems to be a regularly theme - if anything goes wrong they immediately try to say that it's our fault. When we downed the servers over a weekend to increase the memory, they tried to say that caused an issue with the configuration. One of their "basis" consultants had made changes to the development system, but hadn't copied these to the others - of course that was yet another excuse for them to insist we were at fault.


In fact, my guys and I are starting to get pretty good at fixing problems even though we haven't had all of the training that we were promised.

Ah well, tomorrow is another day.

Wednesday 28 January 2009

Partners in crime

One thing that really annoys me about SAP; their use of the word "partner".

"Thats a problem for your hardware partner". "You need to refer that to your software partner". "You should call your FI/CO partner" - "you need to discuss with your MM partner".

Essentially, the use of the word "partner" is designed to make people feel that it is a collaborative exercise - of course, it is nothing of the sort. We are THE CUSTOMERS!! You are THE SUPPLIERS!! Are you listening Walldorf? (Of course they are not - they know better than us miserable creatures)

And of course, it is used by the consultants to avoid actually doing anything. If they can pass responsibility onto someone else, you bet they will.

When SAP was just bought by big organisations, it made sense. Most of these types of company would not do everything in house but would have links to specialists that would take on certain functions. Of course, for smaller businesses, that is not the case. We don't have "hardware partners" but suppliers - in many cases, you buy from the cheapest supplier at the time that you need something. It's not too practical to have long term deals as we just don't buy enough to make it worth while for the bigger vendors.

Equally, it's unlikely that companies of our size would have an "installation partner" - we do 2 or 3 servers a year, maybe 10 PCs. These days, equipment comes pretty much pre-configured - join it to the domain and away you go. But no, SAP want us to discuss with our "installation partners" how the system should be set-up. I then set-up a dummy company to handle this and passed details on to the consultants; they then sent some stuff through to the fake email address to advise me that I need to go on numerous courses to get myself "SAP accredited" as an "installation partner". In fact the consultants were our "installation partners" as they wouldn't let anyone else do it.

This came about because before we signed anything, they were asked what specification of hardware was required for the servers. The response was really vague; so I checked out some very nice equipment, (about $60,000 worth), put the details onto a letter and sent it to them with a request that they could confirm it would be adequate. Eventually they did (verbally) agree it would be OK. So we bought it.

However, about 4-5 months into the project we noticed a major problem; shortly after that, we found a couple more - we had some major system instabilities. They immediately told us that what we had was insufficient and that we had to refer back to our "sizing partners" to find out what equipment we really needed.

I then carried out an exercise with one of the largest suppliers of hardware in the world - they asked for information that we just couldn't give them - until the product goes live, we have no way of knowing what levels of some items we will see. However, I made up some numbers that sounded a bit excessive, and let them have that. The upshot of all this was that we were advised to put some more memory in the machines.

But that didn't fix it, so we were told once again to go back to our "sizing partners". God, if we were told this once, we were told this 10 times - and each time we told them that we had, and that we had put extra memory in as required. Later, they started telling us that we had to put even more memory in - but we currently have 20 GB in each server which should be enough by any standard.

They even started getting really snotty when they talked to the directors, telling them that I was being obstuctive - our CEO had one of their guys in and roasted him. After that they got one of their people in support in Germany to connect and he pointed out that there were some configuration items in the software that were wrong; once these were corrected, most of the instabilities vanished. We asked why these han't been addressed by their people - after all, they did the installation of SAP. The response - each system is different.

Going now before I have a coronary.

Tuesday 27 January 2009

Let's get SAPpy

When we actually started the work on the implementation, it was agreed to have a project team of senior managers to cover the main functions within the business. This makes a lot of sense - however, I was bit concerned that they didn't want to get input from the staff. After all, these are the people that are actually going to use the product on a daily basis - they will often have a far better idea of what work they do and what problems they face than the managers.

As I have studied Requirements Engineering, I had previously carried out an elicitation process to determine some of the needs of the business. This data was presented to the consultants, but it was clear that they had no interest in that information. Their arguement was that they provided "best practice" in operations and would ask us to examine our processes to make them more efficient. Now I understand this view and don't entirely disagree with it; it makes sense to streamline processes before the new system goes in, as it won't happen after.

However, I would suggest that what they define as "best practice" is nothing of the sort. Basically, they have written some variations in their software for various large organisations (who have paid a great deal for this) and they take various bits from different companies that they think are a close match for the business that the customer business is in, and install those. They then try to persuade you that what they have provided is the "best practice" for that function. In some cases this may be true, but from what I have seen so far, the processes are generally clunky, long winded, and very inefficient.

Anyway, so away we went - meeting after meeting after meeting. Death by PowerPoint. I made a point of taking notes at all the meetings I attended; subsequently, I typed them up and published them on the company intranet for all to read. I noted that at several of the meetings, we had more than one consultant present, but they seldom took part - all of the talking was done by one person. They took no notes at all - their agendas were often just 2 or 3 bullet points, and they never published an action list. (I also noted that the spare people were busy emailing other customers of theirs; but we were paying for their time!)

Later, I questioned some decisions that they took and they insisted these had been agreed at meetings. I showed my notes and asked for theirs - of course they had none. They objected to my typed notes saying that they were invalid - I presented my hand written notes to back up the more formal ones. Needless to say, this went down like a curse in a church. They subsequently insisted that I should not be involved in any meetings - I was not then invited to attend any more. In fact, I later made a point of sitting in on a couple later and the consultants' director objected to this - he told my company that my presence was "disruptive".

They referred to this initial stage as the "blueprint" phase - I understand from others that this is a common process with SAP. It's portrayed as providing a proof of concept for the developed product - but they don't actually provide the software at the end of it which seems strange to most developers. What you get is - you guessed it, a series of PowerPoint presentations which indicates the processes from a high level. But as many have found, the devil is in the detail.

SAP part two

So, just under 2 years ago, we signed up for a SAP implementation with the consultant partners that had been providing the initial presentations.

Having seen the contract that was provided for us to sign, I started to get a bad feeling. I'm not a legal expert, but there is no question that what we were given was poorly put together. It looked as if it had been created from a series of other documents almost at random. I did take a copy and showed it to a solicitor - he was astonished that anyone would actually sign such thing. But we did.

A director from the consultants then had several meeting with our directors; a time table was proposed by him and he assured everyone that it would be possible to implement the software within a time frame that he put forward. I suggested that what he proposed was rather ambitious; it was asked if any of their projects had come in on time and to budget and he very confidently told us that they all came in within a 10% margin (which I would say is not a very bad position if true). However, despite numerous requests, we were given not one name of a customer of theirs that we could actually speak to that had successfully implemented SAP.

He also quoted a number of figures for various costs and suggested payback periods. It was pointed out that his figures literally did not add up - the FD did some quick calculations in the meeting and their total was out by over 20% just on the actual figures on their documents. As of the beginning of 2009, we have already spent over twice the figure that they gave us and we are no where near finished.

My name had been put forward to be project manager based upon my experience, but it was decided that a director would act as project manager. He is pretty efficient, although not especially good at communicating with staff - I spent some time trying to put processes in place to help manage consultant time, costs, availabilty of equipment and resources. I bought a number of books that I thought might help with the various aspects (and they were not cheap; roughly $3000).

The consultants were not happy with this - they wanted their own project manager to be in charge. I had suggested that we should have a process where the consultants would advise us when they were to be on site and what they would do with their time; this would then allow us to keep a tight eye on things - they totally ignored this and refused to provide any advance information.

A project was organised to make sure that everyone knew what was happening and why - a team of managers was assembled and some more PowerPoint presentations were provided by the consultants for us to use to help explain the process. However, these items were no better than anything else we had seen - at the end of the initial launch meetings, I was approached by several of the managers who were trying to find out just what was actually involved and where they fitted into the project as they were completely baffled.

It would also be appropriate to point out that these first meetings did not take place on the days that had originally been scheduled, but a week late - and subsequently, the consultants have completely failed to meet a single milestone.

Lets start at the beginning

I did say that I wouldn't do this, but..... I work for company that is implementing SAP and it is a really painful process. I need to get some things off my chest as it is becoming a real nightmare. I'm keeping the blog anonymous as I rather like my job and want to keep it. I don't suppose anyone will actually read this, but you never know.

First things first. I've been in IT for about 20 years. I have a degree in computing from a well respected university. I'm also MCSE, CNA, CCNA, an accredited Prince 2 practitioner, as well as having a number of other qualifications. I'm not a programmer but have had some experience with BASIC, C++, C#, .NET and some other bits and pieces.

The company that I work for has a large number of disparate systems; it is very difficult to get a good picture of what is really happening within the business. It is extremely inefficient and this makes it difficult for us to progress this business, which despite the credit crunch is still doing OK - we have a good product, great service, good staff, loyal customers and it seems likely that we can ride out the difficulties.

However, the decision was made some years ago that we needed to look at an ERP (Enterprise Resource Planning) application, to tie all the different aspects in together. This makes sense; such a package can help us to understand what is going on and provide a means for better control.

Initially, we looked around for various products and came up with a short list. However, a VP of one of the business units decided that SAP was the way to go. There were many meetings with consultants from SAP - quite frankly they were appalling.

The meetings consisted mostly of PowerPoint presentations - they clearly didn't want to show the software in operation. When they finally did, the people involved had enormous difficulties. They obviously did not know their own product and regularly got stuck with even the most basic functions. It got worse; at a couple of the meetings they introduced a number of "Technical" experts. These made several comments that were at best contentious, at worst completely false.

I made my feelings known; I was concerned that the product was just too big, too inflexible, too costly and would require more resources than a company of our size could handle. I also provided a large number of studies and information on failed SAP implementions - the FD also obtained advice from some of the most respected financial institutions that all said the same; don't do it.

Despite this, the decision was forced upon us to go with the product 2 years ago. I must admit that at the time I was concerned, but comforted myself with the thought that I would be learning new skills. Like all the rest of the team, I had seen the number of SAP vacancies and felt that this might be a useful additional skill.

---------------------------
I'm going to add more later - I want to do this in small sessions so that I keep things more or less in chronological order.