Sunday, 4 October 2009

Week 1

Well, the first 7 days have gone by, and so far things are not looking too bad. If it carries on like this, then I for one will be quite pleased.

First thing on Monday morning, I was in very early. I'd made a point of getting as many people as possible to logon in last week to make sure that they knew their username / password and that they could actually get access to the production system. However, I suspected that there would be a few phones calls and I wasn't dissappointed! I had about 5 people had managed to forget their password and completely lock themselves out of the system - I was able to talk them thru getting back on. This was fewer than expected, so I was quite pleased.

Later, I had a few questions about access permissions - although we had spent a lot of time working these out, it was clear that there would still be issues after we had gone live. Some of the decisions proved to be a bit optimistic and it was found that we needed to give some of the lower level staff access to more than had been originally agreed. Again, I expected this, so it wasn't that big a deal for me - but it was for some of the managers, as they want everything done imediately and with SAP that is not always possible.

We've developed a reasonably satisfactory process for getting these changes authorised, but people still try to bypass the process and get very irate when I send their requests back to them with the instructions. However, they will learn eventually. I am getting more than I expected, about 8 -10 a day. Hopefully that will tail off in the next week, but I'm not convinced it will as I'm sure that they didn't check all of the variants prior to go-live.

We had one major discussion - a few weeks ago, there was a request for a change which I said was a bad move as it could cause a potential problem. That request was then reversed a few days later as our Operations guy found out for himself just how bad it could be. But then a week later, he changed his mind again - we needed this to allow the project team to push things thru. Then there was huge problem caused by one of these guys using the permissions inappropriately, so it was taken off - and then added back on again a week later. We've had an SAP Earlywatch report and they identified that this was major issue and that we should remove this permission - so I had to sit in a room and explain to the CEO why the project team had been given it in the first place!

There's one rather odd problem with the Sales documents - for some reason, they are printing an extra blank page in between the normal pages, so a 3 page page document comes out as 6 pages. We've got one of the consultants coming in ths week and it's been highlighted that they should look at this - I suspect that it's a formatting issue, but I haven't had the chance to work with some of these documents so I'm reluctant to play around with it at this stage.

The SAP production system is working well. So far, no complaints about speed. It's clear that some staff didn't get enough training as they are taking a lot longer to carry out a function than they should - again, I suspected that this would be the case. I felt that many had not had sufficient actual hands on work, but it was difficult to fit this in. I'm sure that they will pick up speed, they just need to concentrate on using the product.

One thing that I'm bothered about, is that some staff are still using the old legacy systems. They said that they were just referring to old data, but why would they need to? The same data is in the new system, and is just as available. I think it is a bit of a comfort blanket, and as such, I feel that it is probably OK to leave them alone for the present. However, I don't feel that we should let them use these older systems for too much longer as the data will quickly get out of step.

Anyway, so far, so good. There have not been the more serious problems that I dreaded, so this weekend I've had the chance to relax and take things easier. We are not out of the woods yet, but if it continues as it all did this week, then I for one will be quite relieved and very happy.

Sunday, 27 September 2009

Here we go

Well, that's it. No more prevaricating, no more delays. Tomorrow is the day - SAP go live day. No further hold ups, no excuses. It doesn't matter if we are ready or not, here we go.

The last few weeks have been crazy - the amount of work, mind numbing. Everything else went out of the window apart from the SAP project. All calls from suppliers and cold callers were turned away - all meetings cancelled apart from the project team meetings. And the last couple of weekends, we have all worked our butts off, to do the best we can to make sure we are ready.

There have been a few issues with the data load - it started OK, but then one set failed and it took several hours to find the problem. It turned out that one of the consultants had made some changes in the Development system which had been transported to the Test system but not the Production system. As a result, the data load worked in one, but fouled up in the other. OK that got fixed, but we uncovered some more items like that which delayed the process each time. We started to get a bit worried - in the end, all of the static data is done, we just have a few items left to do, but they are live data items so can be loaded over the next few days. W also found that some pricing information was way out - a change had been made to some data, and no-one knew about it. A minor issue, but it held things up becase we had to find out why.

There was also an issue with some of the product information - the consultants had been asked to make a minor change to allow for a particular piece of data we need to allow us to report on in a particular way - they had done some work, but it was wrong and we only uncovered this when the production manager ran a report. It hasn't been fixed yet, but they say that they'll have a solution in the next 2 days.

There was also a bit of friction between finance and sales - the sales people had cleaned up their data, but left a load of customer information in as they wanted to be able to track marketing effectiveness. The problem was that when finance started to look at it, they found some companies that no longer existed due to M & A activity, plus some that have unfortunately closed due to the economy. I can see the problem from both sides and normally I don't think it would be that big a deal, but everyone is now getting very tense, so smaller issues get blown up out of proportion.

The consultants got a guy to do a check of the hardware and configuration to make sure it's all OK. He sent thru some suggestions for parameter changes which I did on Friday. I checked and all was fine - but first thing Saturday, I got a call to tell me the system had fallen over. I quickly restarted and it seemed all right, but thought I should go in to be safe - a good job too as it fell over again. I looked at the log and realised that it was referring to a problem with one of the parameter changes, the value they had given me was totally out. Having put it back to the way it was, everything carried on ticking over.

The project team have been doing a lot of data check and testing of processes, far more than the consultants said we needed, but not as much as I think we need. I'm not convinced that the guys have been able to test all of the different scenarios or variants - but there's no time left to do any more.

My eldest was invited to a party for a friends birthday - her father was throwing a barbecue, so after I finished yesterday, I dropped by. It was really busy and it was nice just to sit with a beer with the sun going down and not have to think about too much. However, I got talking to to guy that used to work for a company that installed SAP a few years ago. We compared stories and although it was an earlier version, and different consultants, it was a strangely familiar tale. I am glad that it's not just us - it may sound crazy, but it helps when you look at something that's wrong and you can feel that other people have been here before.

Any way, there we are. In total, we have spent just over 3 years on this project, just under 30 months of that with SAP. I haven't seen the final costs and I know that the FD is holding back a couple of invoices and he says he won't pay them to make sure that we don't get left without access to proper consultant support (although I suspect we might be better off going elsewhere). However, the project cost seems to be a little under $1.5 million for the consultants and software - we think that with all the work our people have put in, we have spent another $750,000. Has it been worth it? Only time will tell.

Going to get a good nights sleep - early start tomorrow. I'm not sure how I really feel about it, it almost seems to be unreal. I really don't know what to expect - I'm hoping it will go well, but just a bit afraid that it won't. Still it's too late to worry about it know, just got to get on and deal with it the best we can. I think that the coffee machine is going to be working overtime this week.

I'll try to post more over the next few weeks so that you can see how things turn out.

Sunday, 13 September 2009

Data loading

As you might guess, we are in the process of loading data into the production system, ready for the final tests before going live. The last couple of weeks have been frantic - we are just over half way there.

There have been a few issues - we had a big problem because the consultants hadn't transported a configuration change. The data wouldn't load and we had to wait for the consultant to process the change before we could carry on. There was another open issue for which they finally came up with an answer only for us to find that we had to modify the data load file in order to deal with this new item.

There has also been an issue with one of the tests - it's producing the wrong output figures. We are not sure why as in the tests in the test system, it worked fine. We are still checking this one out.

The guys at one of our other sites has also hit a snag. They wanted an output document for inventroy control - long term, they want a barcode on the paper so that we can use a scanner. Unfortunately, the document that they specified is producing completely wrong data; again we're not sure why.

In addition, there has been a problem emailing paperwork to customers / suppliers. It worked on the test system, but for some reason not yet discovered, it doesn't work on the production system.

Everyone is putting in a lot of overtime - we are all getting very tired and starting to make stupid mistakes. I think everyone needs to take a vacation after this.

Saturday, 22 August 2009

The final stretch?

A couple of weeks ago, I had some contact with a guy that has been in consulting for a long time – he was kind enough to say some goods things about my writing, so I’ve decided to return the favor. Catch his blog here: http://sapsearchlight.blogspot.com/

One thing that I did find of interest; he referred to a process of project management that is supposed to be used by the consultants – ASAP (AcceleratedSAP I believe it stands for) also known more recently as Focus ASAP / ASAP Focus depending on where you get your info. He said “Outside of IBM and Accenture, all certified partners MUST adhere to SAP’s ASAP methodology, sometimes referred to (recently) as Focus ASAP. Most of these partners add some of their secret sauce to the core SAP methodology. I am willing to bet that if you look at this firm’s proposal of services to you that they make a big deal about their methodology.

I did actually look thru all of the paperwork from these consultants – nowhere does it a make a mention of this. I asked around our project team to see if anyone had heard it cited, and the general answer was a definite “No” – apart from one person who remembered reading a reference to “Focus ASAP” in one of the SAPpress books that we bought a ways back. So I then thought I’d ask their Project Manager – unfortunately, he’s “not available” at the moment (I don’t know why) and we are not sure when we will next see him. I approached one of the other consulting staff and asked the question – the response was along the lines of “Oh that was used about 10 years ago, but no-one uses that anymore, it’s a really horrible system”. Interesting?

-----------------

As you may imagine, things are hotting up. For many people the project is starting to get real and you can sense the panic growing. I’ve had numerous project members and departmental heads speak to me and even a couple of the board expressing their concerns. We have no formal contingency plans other than those that I and my IT staff have discussed. We are committed to go-live and nothing will stop or delay that now.

A major problem was uncovered in the last week that has created a lot of bad feeling. As I’ve indicated, we trade across borders so use multiple currencies. We suddenly found that all of the customers had a default currency set and that this overrode any attempt to set the right currency. The initial suggestion was that the data was wrong – a quick look at the files showed that wasn’t the case. So we were blamed for loading it incorrectly, but that was quickly disproven.

Eventually, it turned out that the consultants had set the appropriate configuration in the test system, but not in the production system – thank goodness we found it before we started issuing invoices. They blamed us first for not raising it as an issue – however we did, over a year ago and could prove this by referring to the list of issues. They then said that the various departments hadn’t tested the process correctly – they had, but obviously not in the production system. No matter what we showed, they still tried to maintain it wasn’t their problem. And I’m betting that after we go live, all of the problems we then find, they will say the same thing, no matter what the problem.

In the past few weeks, I’ve been doing some travelling to the other sites (partly why I’ve not been adding to the blog) and I’ve heard of some other stories of SAP implementations going on at the moment. What is really surprising is that they are being marketed as successful, and yet from what the people on the ground are saying, they are anything but.

In one case, its being run by one of the Big 3, the others are smaller operations with smaller consultancies. Not one has come in on time, not one on budget, not even close. In each case, it seems highly improbable that the organization concerned will ever see the ROI that they wanted or were promised. At the biggest one, they went live last year and they were expecting some significant reductions in staff because of efficiencies, but instead they have had to take more staff on in order to manually process items because the system isn’t working right. And in that particular case, they’ve outsourced the data center which they expected would save them nearly $1 million a year – the local paper has found that the outsourced costs have increased their bill, not decreased it.

Well enough for now – I expect that over the next few weeks, I’m going to be very busy, so I may not have the time to keep you updated on progress. However, I may try to post a few short items if I can just so that you see how things pan out. What ever happens, many thanks to you all for reading this and for the great comments - they have been really helpful and I hope that I can return the favor one day.

Tuesday, 4 August 2009

I've got a bad feeling....

Things at work have started to get very tense - people are now starting to realise that the go-live date is getting closer. There are still so many items that have yet to be fixed up, and despite all the effort, it seems that many of these will not resolved in time. So the decison has been made that we will go ahead whatever the cost and live with any issues.

No doubt, many of you that read this will have heard similar stories to the ones that I have heard - about projects that suffer problems. I see it almost like a slow motion train wreck, and there is nothing that I can do.

The PM from the consultants was in the other day - he was supposed to be spending time with the finance people to go thru the closing of period / month / year. When he discovered that they didn't have the right transaction code, he didn't try to resolve the problem or ask me to deal with it - he just logged on as himself and added SAP_ALL to their user accounts. For those that don't know, SAP_ALL is a profile that when it is added to a user account gives that user full administrative access to the whole system.

I found out about this a few days later when I was making yet more changes - to my horror I found that some 25 accounts had been modified to give them the same access level. I've been told that this is needed because they no longer have time to waste and need to get these processes working and that we will fix it all up about a month after go-live. Call me cynical, but I suspect that won't happen and I'd give 50 bucks that they still have the same access level at the start of next year.

I also spotted something earlier - we have yet another new consultant on site to address a couple of specific issues. I wasn't told about this until after he had been there 3 days. I queried how he had been doing any work and was told that he was using the account of one of the other consultants - in addition he has created duplicates of the accounts our project team members use, and he has been using these to test things; but as they haven't been added to specific roles, he just added SAP_ALL.

In addition, it was highlighted that another consultant has been modifying some items - he has made a mess of it and now that item doesn't work at all. They've gotten someone else in to address the issue, but it appears that it will take 3-4 weeks to fix.

There was a review meeting earlier - as we went around the table, it was identified that one of the senior managers wasn't there as he is away on holiday. A comment was made that it was unacceptable that this key person should be away at such a crucial time. It was pointed out the person concerned is on his honeymoon - it was booked up 18 months ago before the date of the go-live had been put back so many times. In addition, the person concerned has had very little training - I think I am the only one that has spent time with him at all (altho I might be wrong). Certainly, he has actually had very little input into the project.

What happened then was astonishing - I tried to point out that this was a bit unfair and one of the directors started shouting and threw his notepad across the table at me (I suppose that I should be greatful it wasn't his laptop - no doubt I would have had to repair it). The rest of the meeting was extremely uncomfortable as you might guess.

I could post more, but just don't feel that it's worth while. I suppose that when it goes belly up, then they will be demanding that I wave my magic wand and fix it all in 10 minutes. I made a point of reading Rudyard Kiplings "If" - I think that it has a lot of resonance for those of us in IT, even tho it was written some 100 years ago.

http://www.kipling.org.uk/poems_if.htm

Sunday, 26 July 2009

Busy, busy, busy....

I was going to write something last week but just couldn't make the time. It's been pretty hectic this week as well.

Part of the problem is that we have to do the SAP work at the same time as our normal tasks. We have to maintain hardware, patch software, keep the AV running, check backups, hold people's hands when their printers don't work etc. In addition, we have a 5 year development plan that gets updated every 6 months or so and we try to make sure that we stay on track. Over the last couple of weeks, there have been some changes to buildings and we were moving hardware around, doing some cable patching etc. This is straight forward if it is planned out - however, some of our managers think forward planning is deciding what we will have for brunch!

The SAP data loading has started - I've made the point to people several times that now this has started, we can't really delay the go-live date. Of course if necessary we could, but it would cause a large number of problems. However, we want to concentrate their minds and make sure that they understand the importance of the process. Despite this, some still haven't gotten with the program - we loaded some price information only to be told a day later that they wanted to make some changes!

Now I did the intial tests of the loading process way back and setup the scripts to import data. One of my staff was put in charge of the data assembly and he's done an excellent job. He's also tested the script in the test client to make sure that it should all run OK. So when we started the load process in the production system, we expected it to go well - and for the most part it did. Unfortunately tho' we hit a couple of snags.

For those that don't know SAP, you have 3 systems, Development, Test and Production. In those you have "clients" - effectively instances that can be run separately. One of these is designated the master client and all changes are made in there - then a system of "transports" move the configuration changes so that all systems are the same. It turns out that several of the consultants have been making changes to the configuration, but then haven't transported it to the production client correctly. Worse, they didn't even put the changes in the master client, so any transports will over write what they've done. We've had to go thru and sort these out as we find them, which is slowing down the load process - so lots of late nights and weekend working.

On top of that there are still a large number of outstanding issues, and one of these is to do with the finances. Apparently the consultant for that area told the project manager that he was intending to deal with it after go-live. I wish I'd been there to hear the rest of the conversation!

We still haven't gotten the labeling issue totally resolved and the PM wants one of the consultants back on site to deal with this. They've complained that they wanted to deal with this a few weeks ago, but couldn't as we were dealing with other matters. They can be a real PITA at times - we have to wait what seems an eternity for them to get back to us, but when they finally do, they expect us to leap into instant action at a single word.

I'm still waiting to find out what they intend to do with the CRM - it still runs very slowly and there are numerous parts to it that don't meet our needs. The sales director has indicated that if they don't get it fixed soon, he will make the decision to toss the project. As far as he is concerned, he would rather that we use the existing product as that works. We can't link it to SAP, but that doesn't bother him.

Several of the project team have started going back over the training with their staff - I've been suggesting that they should do this for months as most staff have forgotten everything they were told. There were a couple of sessions last week - of 19 staff in training, 3 could remember their password, the others all had to be reset! However, once they went thru the basics again, they stated to recall some of the steps. Unfortunately, I don't know if we are going to be able to get everyone back in the training room. We'll just have to wait and see.

Onward and upward!

Thursday, 9 July 2009

And it's all gone quiet

I haven’t posted for a couple of weeks – well, that’s mainly because not a lot has been happening. So I thought that I would go back in time a bit, and cover some of the main issues from the early days in a bit more detail.

We had a launch meeting over 2 years ago now (unbelievable how the time has flown) and it had been agreed that at that meeting, the consultants would discuss the various steps of the project, and it was expected that we would come away with a clear idea of what had to be done, who was to do what jobs and have a timetable that we would all work to. Well, that WAS the plan.

I still have my notes from that meeting – one of my first points was that the agenda that they put up on the screen was not the one agreed with the project manager. In fact, it only had 4 bullet points on it – nothing else at all. When you consider that this was for a meeting that was expected to last all day, I felt that it was a bit light on detail.

I also made a couple of other comments after about 30 and 45 minutes – “when is he going to get down to it?” You see, the meeting was being managed by the director from the consultants, and he was doing most of the talking. That first hour, he did a LOT of talking – but at the end, I had no notes as he had not covered a single item that was actually relevant to the project plan. I could see that a couple of our senior managers were getting restless and the CEO left twice to take phone calls.

We broke for coffee at 11:10 – my real notes by that time were on 5 lines although I had some real nice drawings, none of which had anything to do with the project. During the break, our CEO and the consultants’ director got together and I’m told the CEO told him to move the pace up a bit. But it has to be said that by lunch, we still hadn’t really any better idea of what the project plan was. My favorite part was when the guy turned to the CEO and suggested that we ought to have a few bottles of champagne to get the project off to a good start – of course I had to speak up and suggest that perhaps champagne might be more appropriate once the software was actually running. Funny that he and I have never got on since.

After lunch, we were back to this guy talking on and on without actually making any real positive comments that would have been of any value. Later, just before the afternoon coffee break, he allowed their project manager to say a few words. This guy did actually discuss the timetable – but took no more than 20 minutes. Essentially, he stated that there were to be 6 key milestones (there were more, but he only referred to the 6) and he gave the dates for those, one of which was the meeting we were at (10 days late already) and the last one was for the go-live date.

By the end of the day, all we really had was a very basic timetable showing the start and finish with the “blueprint” phase, acceptance of this, the data cleansing, data loading and cutover to the live system. The idea of data cleansing had been briefly covered, but in very limited detail. I had suggested that we should start this as soon as possible as I knew we had a lot to do – this was brushed aside and in fact it didn’t start for a further 4 months. I should highlight that we are now some 18 months on and today it was found that one particular set of data still hasn’t been corrected and won’t load in its current state even though it has been sent back to the relevant department 5 times.

It was a further 2 months before most of the project team even really began to know what was expected of them; none of their responsibilities had been discussed and nothing was down in writing. I know that I can be a bit anal, but for me this so important. People have to know and understand what is expected of them – most won’t make notes, so I feel that writing it down so that they can refer to it is a must.

If I were to ever be involved in something like this again, I know that my priority would be to make sure that the launch meeting set the tempo for the project – that within 30 minutes, everyone would know exactly what was expected of them as individuals and of the team as a whole. We would have clear definition of responsibilities written down, with due dates and standards expected.

I think it a shame that the experience that I have gained from this project will never be used to make another project more effective – that is partly why I’ve been writing this blog, as I hope that other people might come across it and find it of value. The skills that I have gained have come at a high price in terms of workload and frustration and it should be possible to help other people avoid this. Certainly from what I have seen, the consultants have learnt little from it and are most likely to repeat the same errors, over and again.

Oh well, back to the grindstone.