Monday, 3 October 2011

Have we been here before?

So - we were supposed to go live at our overseas site today. Up until Wednesday of last week, they were still determined that it would go ahead, come what may. This was despite no more than 10% of the data having been loaded, and we are still waiting for quite a bit of data to be extracted from their legacy system.

I only finally got the confirmation of the delay midday on Thursday - it's been put back to December at the moment, but I'm not sure if that will stand either. I know that the site manager is relieved as he was really concerned that they are just not ready, and I believe that most of the staff there are equally pleased. Our project team are not entirely happy - but they all accept that it would be too much of a risk to try and go ahead.

Now we have a bit more time to try to resolve some of the issues that are still outstanding - but I'm a bit concerned that we may end up with a few more as well. There have been a couple of config changes made and once again, it has caused us some problems. (Did someone mention testing?)

This time it's the purchasing, which is a real pain as that has actually been pretty good for some time now. It appears to be to do with the release strategy - the change means that staff can release purchase orders to any value, instead of the staged release which took us about 3 months to get right. I left the manager in charge of the purchasing department working his way thru a number of items to try and get a handle on exactly where it has gone wrong.

The consultants fixed the printing problem on the purchase order - well almost. There's a bit of an issue with the vendor addressing and a number of the POs have completely the wrong details. I got a message earlier today to tell me that there is a specialist going to take a look - it appears that he has already had access to the system, but he's been using someone else's account. I've now created an account for him, and asked him to use that - I've had no response, so I don't know if he hasn't received the message, or just can't be bothered to do so.

There was also an issue on the finance side (not quite sure what it was) and they had arranged for a consultant to look at the problem - we were told that it would take them about 4 / 5 days. However, the FD was looking at it, and with the help of a Google search, he has fixed the issue. The SI are not pleased at this.

Talking of the SI, I've had one of their people chasing me to outsource the systems to them. This is something that I'm not particularly keen on for a number of reasons. They insisted that it would all be managed in a facility in this country, but when I checked, the data center they use is actually somewhere else in the world - I think it might be either eastern Europe or Malaysia, I've not been able to clarify.

Their sales guy insisted that it would be cheaper to use them - well there would be savings in hardware / software / utilities, but I doubt very much that it would work out any cheaper. Some time ago, I looked into the costs, and based upon what we would need, the price from the only hosting center prepeared to give me an estimate would indicate that we would be looking at $300,0000 a year now, rising to $600,000 in about 3 years time. I can tell you, that would be more than our current IT budget.

However, it's not just the cost that worries me - this system is going to be the key system for the whole business. A decade ago, it wouldn't have mattered if it went off line for a few hours or even a day or two - that's not the case now. Quite simply, a loss of service of more than a few minutes would be problematic, more than a hour could be disastrous.

Based on their performance, can we trust our key system to them? Of course, it's not just the potential loss of service. I don't feel that they have demonstrated that they work in a particularly secure mananer, and I'll be honest, the thought that they might lose data scares the heck out of me. However, it won't be my decision anyway - this is something that the board will decide. Apparently, they are arranging for a junket at a country club for the shareholders and the board, to advise of them how it would work - I have not been invited.

However, I have been asked to put together a presentation of my concerns, highlighting the costs. I've also included some statistics on what we have achieved and how much we have saved by doing it all inhouse. In addition, I've put forward some plans for future development that would allow us to ramp up the resources to deal with required increases as the other sites come on line. We'll see what happens.

Back overseas again next week. I'm really racking up the frequent flyer miles - next time, I'm taking my wife with me!

Sunday, 25 September 2011

I'm Back!

I’m back! Before you ask, I had a great time thank you.

Unfortunately, the flight back was delayed – and we got home very late on the Sunday. There were a number of messages waiting on the ansafone (I hadn’t taken my cell phone), and essentially, the messages all said that I was to get over to our overseas site asap. So I packed a fresh suitcase, and got my flights booked – with an early flight, the time differences plus the delays the day before, I was really very tired by the evening.

I spent the week over there, and then it was decided that I should go back again this last week. What with that plus trying to do my normal work and various other chores, this is the first chance I’ve had to sit down and analyze the situation.
I’ve not been told officially, but it appears that a number of staff have been told that the overseas site will not be going live in October after all. I’m not surprised – the data load was supposed to have started the 1st week of August and it hadn’t started by September. Even now, very little data has actually been loaded, and it appears that there are still some unresolved issues with parts of it.

During my first week over there, I spent a bit of time doing some training – about half of the staff there had still not even logged onto the test client, and they are a long way from being confident in using SAP. I also had a very upset General Manager complaining that there was a printing problem. When I checked, it was actually only one document that had the problem – the sales order form, so rather important that they gets this fixed.

I haven’t done that much work on the smart forms, but I took a quick look, and discovered that one their consultants had made the change. Unfortunately, no-one knows why or what for, and the guy isn’t available. I’ve left it to them to chase up, but so far, it’s still not printing at all.

Another issue has also appeared recently that affects the financials. At first, the consultants suggested that it was an authorization problem, but we’ve now identified that certain products are being assigned to the wrong ledger code. Again, no-one is really quite sure why – we have our consultants telling us that the configuration was correct, and their consultants are saying that it’s not. So they’ve made a change and now we have a problem.

Our Accounts Manager is really upset about all of this. A couple of times now, she has spoken to me in private, and it’s clear that she is very distressed about the whole scenario. She’s been with the company a long time, and has been thru some really tough times, but I think that this is the worst she has ever felt about her job. I just don’t know what to say to her anymore.

I’m also a bit concerned about the amount of testing that they have done. If you’ll remember, a bunch of us went over back in April, and we did some full end to end testing of processes and made sure that everything worked – but that was only using a limited amount of test data.

The guys on site have access to the test system and although not all of the live data has been loaded, about 80 – 90% is available. But the testing that they have been doing is not the same end to end testing. Yes they have put a whole ton of quotes, purchase orders, sales orders on the system, but they have not then followed this thru to make sure that orders appear on the MRP or that sales get billed.

It could be argued that we shouldn’t need to test again, as it was proven to work in the development system and we are using the processes in the production system. But the issue is that their consultants have made a few changes and I strongly feel that we can’t be absolutely sure that any of their tests would actually work unless we do try it out.

I did manage to speak to the GM of our overseas site one evening over a beer – he is very much of the opinion that they will not be ready for the start of October. He did suggest that perhaps they could have a staged approach to the go-live, perhaps by working in tandem on both systems. I wouldn’t have a problem with that in principle, but they don’t have that many staff, and I’m not sure they could actually do that.

Oh well, back in to work tomorrow, and maybe I’ll find out a little more about what the plans are.

Saturday, 20 August 2011

Problems with prices

About 4-5 months after our go-live, I received a request for a change to a role. At that stage, we were still making quite a few changes as we discovered issues that had not been uncovered during the testing phase. This was quite normal, but something about the request didn’t seem quite right.

The particular role was one that I had worked on with the sales manager, and the request from our second site seemed to indicate that there was a problem that should have affected everybody. When I went into it in more detail, I wondered if it was due to the staff concerned not entering the correct information.

I passed this back to the relevant key user from sales, and she confirmed that the process was working for each site – it seemed that this was an issue with staff training. So the sales manager organised a quick refresher session with the sales staff at the second site.

Whilst he was going thru this with them, he also spotted an issue with the cost price of one item that they were using. Quite simply, the cost price shown in SAP was way out – in fact, even with the price uplift, they were actually selling the item at a loss of about 25c each for a selling price of just under a dollar. He then started checking a few other items – most were OK, but there were more than a few with a price that was wrong.

Subsequently, they spent some time going thru the rest of the price list to make sure that this was OK – and based on what he found, the second site had lost just under $10,000 in incorrect pricing during those first few months due to an incorrect price having been loaded during the data load process.

An investigation proved that the price had been wrong in our old system. At some stage, the price had gone up but never been altered in our old ERP system. No-one knows how much we lost in total, but most agree that it’s probably over $25,000 for each of the previous 4 years – a total of at least 100 big ones. You can imagine that there were some people that were very unhappy about this.

The reason I raise this now is that we were doing the training with our overseas site just over a week ago, and it appears that they also had the same incorrect prices in their old system. This has caused some real upset as people are trying to uncover when things went wrong and why. I doubt that we will ever know.

Of course, one thing we can say is that is not down to SAP. The problem occurred long before we started the implementation. It could be argued that this should have been picked up at the data cleansing phase, and that’s probably true – but it wasn’t. No matter what anyone thinks of the software, there is a limit to how much it will cope with the failure of the human element.

So chalk one up for SAP – while the correction of the price won’t cover the costs of the software, it’s identified a major problem that has taken money off the bottom line, and one that would quite possibly not have been discovered for many years. Although there are still questions about getting an ROI for the project, each of the areas where it has made a difference will help to justify the decision to buy it.

Oh well, vacation time is here. I feel that it is overdue, and am looking forward to a good break.

Saturday, 30 July 2011

None so blind...

I started to write this item a couple of weeks ago – but didn’t finish it at the time. I was highly annoyed and wanted to think about things before posting.

We actually started work on the SAP rollout at our overseas site almost 2 years year ago. However, nothing was ever completed, even in the development system and eventually, the project stopped before being started again with a new set of consultants in the middle of last year. The plans then were to go live at the end of October last year – as the previous consultants had done quite a bit of work, it was felt that this was achievable.

However, it quickly became clear that most of what had been done was of no value. Essentially, it all had had to go right back to the very beginning. The SI were still trying to say that October 2010 was achievable, but most people on our project team could see that was never going to be possible. A date was set for the end of July this year – most people agreed that this could be done if all went well.

As it happens, in April of this year, the consultants still had not finished working on the development system. We also had issues because almost no testing had been carried out, and absolutely no training had been done either. Our project team then went to the site to do some work, and as I’ve indicated, we managed to get a great deal done. So much in fact that we all thought it might still be possible to go-live in July. There is still major work being done by the consultants, but from experience, it seems to be normal for them to still be working on items right up to (and even after) the go-live date.

But there has been a major problem with getting data loaded. First getting data out of their existing system has proven to considerably more difficult than anyone believed would be the case. Cleansing the data was also a bit of an issue – but the biggest problem has been getting it matched to the relevant data load into SAP. I had been told by the SI that about 80 - 85% of the data load had been completed into the development system – but during our test phase it became very clear that it was really closer to 10 - 15%.

Since then, some data has also been loaded into the test system – but no more than about 30 – 40%. Some training has been done with key users, but none with the rest of the end users. I had to go over there again a few weeks ago, and they were trying to do some testing work in the test system. Most of this failed simply because there was insufficient data to allow more than few test scripts to be used – it took them as much as half a day to complete a single test item.

As it became clear that July was slipping away, a decision was made to move the go-live date back yet again. This makes sense, and I have no issue with that other than once again, we are committing many resources to the project than was originally planned. But what has outraged me is a decision that has been made following further discussions with the SI.

The director of the SI suggested to our senior managers that we did not need to complete the data load process in either of the development or test systems, but simply go straight to the production system and load the data without having to do any testing. The argument was that we had done a few items so that proved the process worked and therefore any further data added to the same process would also work. He also argued that if we needed to have data in the other systems, it would be easier to do a system copy after the go-live.

I tried to point out that this view was completely wrong – we have seen repeatedly that people make mistakes when preparing the data. In some cases, the mistakes are so significant that the data will simply not go into the database without considerable manipulation by the people carrying out the work. To trust that all of the data will load correctly just because you have loaded a few lines is at best, wishful thinking in my opinion. As far as I am concerned, completing the data load process in the development and test systems should then save time and effort when loading into the production system.

But there we are – I can do no more than point out the issues and then do my best to see if I can fix the problems when they occur. But I must admit that I am now really beginning to feel that I can do no more on this project. I feel so frustrated all of the time – all I hear is that everything is going well, when it is abundantly clear that things are far from satisfactory.

I have no problem people trying to maintain a positive attitude – I can see that sometimes this is a necessity. When something requires such a major business transformation as SAP, it is essential to keep everyone motivated. But what worries me is that we are seeing “ostrich management” – burying their heads in the sand in the hope that if they don’t see the problem, it will just go away. I can’t believe that will be any good for us, now or in the future, on this project or any other.

Sunday, 10 July 2011

Eye of the beholder

Last week, I had the opportunity to visit the manufacturing plant of another company. These people started to work on SAP a couple of years before us, and I had been told that they had been successful – so I was interested in talking to them in a bit more depth.

I had the chance to talk to the IT manager on site – a nice enough guy, although he seemed a little distant. As we talked, he made the point that they had gone live after just 8 months and they had experienced no problems. I won’t say that he was being arrogant, but there was a definite sense that he was as far as he was concerned, their project had gone very well indeed, and he was more than a bit proud of that. When I told him of our project and how long it took, he made me feel a bit bad.

However, I also had the chance to their production guys, and they had a slightly different story to tell. A few days after go-live, the system had gone down, and it took a couple of days before it was working again. They didn’t know why, all they knew was that it caused a few problems for them. That was bad enough, but it appears the same thing happened on several occasions over the next 4-6 months. Added to that, it appears that they still have an on-going issue relating to some of the data that they are accessing – they couldn’t tell me why.

After that, I spoke to a couple of their finance people. They remembered the first few months as being a complete nightmare caused by a series of issues with billing documents, financial reports and figures just not adding up. It appears that most of these are sorted, but they still recall those problems and again, they are still finding issues on a regular basis.

Later when I checked, it seems that there is quite a difference between the size of the projects, even tho’ the companies are about the same size. They have a lot fewer products than us, and there was a lot less to do on data migration. In addition, they don’t use a couple of them modules that we are using – and these items were seen as being responsible for a good proportion of our delays. Possibly our project might not have taken so long if we had been in the same situation as them.

When I thought about it later, it seemed to me that this is a case that the individual people were really only concerned about what had happened within their own areas of the business, and the project had been judged by that. As far as IT was concerned, the project went well because it came in on time and budget – the other issues were irrelevant. But for the other departments, they could only see the issues that had caused them major problems. Even years later, they don’t see their SAP project as being that successful.

I was once told by a CEO that he liked me because I looked at the bigger picture, and in his experience, most people in IT just didn’t do that. I can appreciate that the IT manager in this case was pleased with the project and felt that his team had performed well – but I could also see that the other departments were not as happy and for valid reasons. OK, they did get things sorted, but it was clear that it could have been a lot worse for them. When I compare that to our project, I don’t feel quite as bad.

I think that with an ERP system, it is important that people look at it from an over view rather than just within their own specific areas. If I think back, it seems to me that when things weren’t going well, it was because it was a single person or a couple of people working on their own without involving others. When things went well, it was because we were working as a team.

I know that the SI have been telling people that our project was a success – and if you look at it from a particular perspective, they are probably right. SAP is working for us, and in some respects, it has achieved some of the primary requirements. From the company point of view, that’s not the whole story – the project has been very expensive and taken a lot longer than we had been told it would. In some areas there has been no benefit or processes are much more difficult – in other areas, we have seen improvement or there is an indication that we may see this in the future.

Essentially, it seems that the quality or success of an SAP project is very much in the eye of the beholder. Personally, I would rather judge on well it achieves the requirements stated at the beginning of the project – but then, that’s just my opinion.

Saturday, 18 June 2011

Paperwork

Right back at the beginning of our project, I talked to the SI about the need to provide us with detailed information concerning what changes they had made to the system. To me, this is just good practice – any time that someone makes a change to a system, it should be documented. There are a number of reasons behind this – people are not always available to confirm what has been done, jobs change and people move, security and audit issues etc. so good documentation is really important, especially before you start making other changes.

The SI promised that we would have full documentation, and they did indeed send us a document once we had gone live. However, it contained almost no actual information on the changes made, other than some flow charts to show processes and was quite a small document compared to the work done. I queried this at the time, and was told that this was standard SAP documentation – when I pushed a bit harder, the guy concerned got really snotty with me. His view was that this was the same format provided to everyone else implementing SAP, and those people had no issue with it, so I should just accept it and stop complaining.

I spent quite a bit of time trying to see if I could find a way to uncover what config chnges had been done, but I must confess that with all the other work going on at the time, I simply could not keep track of what was happening. It also became pretty difficult to maintain details of who was working on what, when, why and what they did. We had an issues list, and for the most part, this is now our primary resource for determining what work was carried out.

As it happens, when we applied a number of support packages, we found several issues. After investigation, it was determined that some of these (not all of them) were down to config changes having been made, and this caused a number of delays until we could complete various work to deal with the situation. It definitely took much longer to do the updates than it should have, and the delays impacted other work.

However, about a year after we went live, we had another consultant on site for a few weeks. When he left, he handed over a really detailed document that we still have – it gave information on what he had done, what changes, what code even. The file was everything that we had previously asked for, and exactly what was needed. It ran to a couple of dozen pages, and was really helpful as we could then refer to it when there was a question about an issue a few weeks after he had gone.

As you can probably tell, I think that this is a very sensible and professional way to work. I raise the subject because there is a particular consultant that has been doing quite a lot of work on our systems, and a couple of weeks ago, we found out that he has now left the SI. The trouble is that he has documented nothing – and I mean absolutely zip. No-one at the SI can talk to the guy, and they have no idea of what he had done, or how far he has got with the work he was doing other than at a pretty basic level.

OK, we will survive – but it is not the best way of working. It’s just another example of bad practice, and when you are paying about $1500 a day, I think that you are entitled to expect a certain level of professionalism and I am not convinced that we’ve seen that from some of the people that we have had working for us on our project.
Having said all that, I would understand if anyone then turned the criticism back on us. The systems are ours, and it would not be unreasonable for someone outside of the company to suggest that it is our responsibility to maintain records of what work is done. As it happens, we do keep some records for various reasons and we can point to a number of audit trails that show we are attempting to maintain a level of control.

The problem is of course that in many ways, the SI and the consultants working for them were working in the way that they want to, rather than in a way that matched what we want or need. I suspect that for most of the consultants concerned, this is the same way that they have worked on all of their projects, and they know no different. Our project leader should have forced the issue (and he freely admits that) but he was taking advice from the SI on how the project should run. But we will probably never know all of the configuration that has been done.

Well it’s too late now – just got to keep going as we are.

Monday, 30 May 2011

No change here...

It’s been about a month since my last post – and there’s a simple reason for that. I’ve been really busy on other projects, not related to SAP. I’m not the only either – several other departments have also been catching up on various things that have been delayed.

We started the SAP implementation about 4 years ago. At times it seems astonishing how quickly the time has passed, at other times it seems that we have been working on it for ever. The problem is that during that time, so much effort has gone into the project, that many other jobs have been postponed. In a couple of cases, this has not been a good thing – we needed to get on and get them done so it was decided that we would do exactly that.

For IT, there have been a couple of hardware refresh jobs, replacing workstations, printers and a Xerox machine. We’ve also been testing a couple of tablets that we think might be of value to the sales force. There have been couple of jobs done within some of the offices to make more efficient use of the space available, so we were also getting involved in the cable work. As you can see, this has meant that there hasn’t been a lot of time left to do more than basic work on SAP.

After the last visit to our overseas site, everyone felt that we had gotten thru a lot of the outstanding work and had almost caught up so that we would be back on schedule. When we left, the point was made that they needed to get on with the data preparation so that it could be loaded. We had done some loading of a few basic data items into the Development system, but not all of the data, and literally only a few items in each category. A decision was made that we would load no more into the Dev system and just go straight thru to the QAS system.

I was a bit concerned about this, as I felt that the actual loading wouldn’t really take up that much time, and felt that it would make sense to try to get all of the data sorted and proven. However, I could see that we need to keep on schedule – I’m not happy about cutting corners, but I can probably live with this.

However, when the data arrived, it quickly became clear that it was simply no good. Not one of the data loading batch jobs was able to complete successfully, and most were so poor that we were looking at maybe 10-20% had gone in successfully on each batch. It was so bad that the project leader took another trip over there to talk to them again. When he arrived back, he was really pissed – he went back over the requirements with their people but feels that they still don’t quite yet get it.
I haven’t been asked if I would go over again, but I know that there have been discussions, and I get the feeling that they are expecting me to offer to go there in the next couple of weeks. As it happens, there is work that I was planning to do, but I’m not sure I will have time to do both jobs, the planned work and talking with them about the data preparation and cleansing.

There is no question that it is essential we get the data right. We found this out ourselves when it was our turn. We have made the point to the people at the overseas site, and several people discussed it with them, making the points very clearly. We provided some sample files showing how it should be done, highlighting the main pitfalls – but none of this seems to have made the slightest difference, and they are now well over a month behind where they should be, and falling further behind each day.

No-one has said anything, but I’m now a bit concerned that they will cut back on the testing in the QAS system before we go live. I’m also a bit worried that staff over there won’t get the full amount of training before their cutover. For me, it would make sense to push the go-live out again, and make sure that things are done right. But it seems that there is something going on in the background, and they don’t seem too keen to discuss any further delays.

Well, we’ll probably know a bit more in a couple of weeks’ time.