On Monday morning, I was on my way to work when I was called on the cell phone asking if I knew that the Production system was running slow. I didn't, but said I would look at it as soon as I got in. While I was getting set-up I had a couple more calls from people to tell me that things were not going well.
When I eventually logged on, I saw that there was a job that had been running since the Friday before - and it was logged as being one of the consultants that hadn't been on site for some time. I asked around and eventually was told that this particular consultant was due to be on site - he had been organised to look at a particular issue.
When they guy actually arrived, he looked thru the system and checked out what was going on. It turns out that he had started this job remotely because he needed to check on some data - he thought it was going to run for maybe 6 - 8 hours. Instead it was still running after 60 hours and only finished later that day. It also didn't do what he wanted.
After he did some further checks, it turns out that a change had been made to the system that would actually cause an issue with the program that he had tried to run. No-one knew about the details as the guy that made the change is long gone, out of the country and there were no references to the work anywhere. In fact the consultant onsite wasn't entirely sure what change had been made. He did say that it had only been done in the production system - there was no transport for the change, and when he tried his program in the Dev system, it ran OK.
It now appears that we are going to have to pay to get one of their guys in to look at this now - they need to try to figure out what was done and then first correct it, and second, try to do whatever job it was supposed to do properly. I asked if the work would be FoC (well it was their guy's mistake) but no such luck. In fact, they don't actually know how long it will take to investigate the issue - possibly 4 -5 days which we will have to pay for before they tell us how much it will cost to fix.
This highlights a couple of points. First of all, most of the work that the SI has done has never really been documented, even tho' that was a point raised right back at the beginning. We were told categorically, that they would document all changes that were made - in fact, we have very little evidence for any of their changes.
Another major issue, is that a lot of the work has never really passed any form of Quality Control gateway test. Again this is an issue that was raised on many occasions, and we were assured that we would have the power of sign off on everything. Yes there have been items that we have signed off on, but there are a ton more that we haven't. In some cases, we don't know that work has been done, and are only finding out when things go seriously wrong, and investigation uncovers the root cause.
I've spoken to numerous people and the documentation that we have been given by the SI right thru the business is so poor as to be virtually useless. We have very little information on a lot of the changes, and it's difficult to see what we can do now other than get a specialist in that can uncover some of these, and hope that we can get them documented in future.
I've also highlighted that they often ignore good practice - and here was another example. A change was made directly in the production system without having been tested in the other systems. I'm also concerned that some of our people have learnt these bad habits - I regularly get asked to do something directly in PRD, and if I refuse and point out the correct procedure, I get an argument that the consultants do it that way.
We should have had a proper change management procedure enforced from the beginning, and I will say that when I queried this, I was told that SAP had a robust procedure that the SI staff would follow. There is a procedure, but the consultants don't bother to use it, and it sometimes seems that they see it as a PITA to be avoided rather than good practice to ensure that we don't get problems.
For me tho' the big issue is that all of these problems are completely avoidable - there is simply no excuse. I wish that I had a way to identify just how much of the money that we have paid has actually be wasted on work that does not do what it should, or has never been tested correctly.
I just hope that we don't get off to another poor week this Monday.
Sunday 25 March 2012
Friday 16 March 2012
Moral musing
Earlier in the week, Greg Smith of Goldman Sachs quit his job. This was a high profile position, and he had a lot of experience, a great deal of authority and was earning a considerable sum of money - yet he felt that he could not continue working there. He arranged for a letter outlining his reasons for quitting to be published - if you want to read it, there is a copy at this link. http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-goldman-sachs.html
I know exactly what Greg was writing about; some years ago, I worked for a company that was doing some things that were morally wrong, and once I realised the situation, I couldn't get out quick enough. But it's always going to be a difficult decision, and I think that he has shown some considerable moral fiber and courage in doing it so publicly.
I have to say that I understand his views and that he is probably doing the honorable thing. I doubt that he will suffer too much - no doubt, he has saved enough to be able to stay off welfare and he may even find that there are other organizations that will be prepared to offer him something as his knowledge will be highly valuable. But it got me thinking about my reasons for doing this blog, so I thought that I would go through them again.
First of all, I will say that when I started writing, it was quite simply to vent my frustration at what I saw was a project that was in serious trouble. Having been a project manager in my past, I was not happy at the lack of communication between the SI and the project team. I could see that work was not getting done on time, project milestones were being missed or ignored, and I had serious doubts about the abilities of the various consultants that were working on the project. Even at that stage with limited knowledge of SAP, I could see that the quality of work being completed was poor at best, and the knowledge transfer was almost non-existant.
However, I am more of a positive person by nature (honest!) and after the first few posts, I wanted to try and make the blog more of an objective analysis of what was happening. I felt that if I could highlight some of the key issues, then others might see these and perhaps this would help them to avoid similar issues in their own projects.
I've kept the blog anonymous - quite simply, if I identified myself, the company that I work for or the SI, I'm pretty sure that before midday of the next working day, I would no longer be working in my current position. I might also find it difficult to get a new job - with the current climate and an ex-employer that won't provide a reference, it would be a bad situation. Although I do have some money put aside, it's no where near as much as Greg Smith - I simply cannot afford the grand gesture.
I will also say that I accept some of my writing is about my opinions - other may look at the same situation and say that things are not as bad as I make out. While I may think that some of the consultants working on the project are not doing their jobs well, that is just my view, and as I can't give them the right of reply, it would be unreasonable to identify them too closely.
I do try to provide an objective view, but in order to keep my anonymity, I have done things such as change dates and sequences of events so that it would be harder to positively identify myself or my company. I don't believe that any of the changes I have made would make a huge difference, but I feel that it is right to make that point clear.
We have an SAP system that is working and thanks to the hard work of the various members of our project team, it is doing a lot of what we wanted. There are still areas that are not entirely satisfactory, and we are working on those - as far as we can, we are trying to build up our internal skills so that we can rely on our own people. If we mess it up, then it will be no-one's fault but ours, and if the consultants want to come back and laugh at our attempts, they are welcome to do so.
There is a possibility that I may come across as arrogant - I take pride in my work, I like to think that I am damn good at my job, and I can point to the fact that the IT in the company is light years ahead of what it was when I first joined them. I've had a number of people visit from outside, and they are generally very impressed with what we have achieved - we are way ahead of the curve for a business of our size. But I don't believe that I have the answer to every single problem, and I am willing to listen to ideas or advice from anyone.
Hopefully, this blog will be a useful resource for some people - if just one person finds something in my writing that helps their project in some way, I feel that it will have been worth it all.
I know exactly what Greg was writing about; some years ago, I worked for a company that was doing some things that were morally wrong, and once I realised the situation, I couldn't get out quick enough. But it's always going to be a difficult decision, and I think that he has shown some considerable moral fiber and courage in doing it so publicly.
I have to say that I understand his views and that he is probably doing the honorable thing. I doubt that he will suffer too much - no doubt, he has saved enough to be able to stay off welfare and he may even find that there are other organizations that will be prepared to offer him something as his knowledge will be highly valuable. But it got me thinking about my reasons for doing this blog, so I thought that I would go through them again.
First of all, I will say that when I started writing, it was quite simply to vent my frustration at what I saw was a project that was in serious trouble. Having been a project manager in my past, I was not happy at the lack of communication between the SI and the project team. I could see that work was not getting done on time, project milestones were being missed or ignored, and I had serious doubts about the abilities of the various consultants that were working on the project. Even at that stage with limited knowledge of SAP, I could see that the quality of work being completed was poor at best, and the knowledge transfer was almost non-existant.
However, I am more of a positive person by nature (honest!) and after the first few posts, I wanted to try and make the blog more of an objective analysis of what was happening. I felt that if I could highlight some of the key issues, then others might see these and perhaps this would help them to avoid similar issues in their own projects.
I've kept the blog anonymous - quite simply, if I identified myself, the company that I work for or the SI, I'm pretty sure that before midday of the next working day, I would no longer be working in my current position. I might also find it difficult to get a new job - with the current climate and an ex-employer that won't provide a reference, it would be a bad situation. Although I do have some money put aside, it's no where near as much as Greg Smith - I simply cannot afford the grand gesture.
I will also say that I accept some of my writing is about my opinions - other may look at the same situation and say that things are not as bad as I make out. While I may think that some of the consultants working on the project are not doing their jobs well, that is just my view, and as I can't give them the right of reply, it would be unreasonable to identify them too closely.
I do try to provide an objective view, but in order to keep my anonymity, I have done things such as change dates and sequences of events so that it would be harder to positively identify myself or my company. I don't believe that any of the changes I have made would make a huge difference, but I feel that it is right to make that point clear.
We have an SAP system that is working and thanks to the hard work of the various members of our project team, it is doing a lot of what we wanted. There are still areas that are not entirely satisfactory, and we are working on those - as far as we can, we are trying to build up our internal skills so that we can rely on our own people. If we mess it up, then it will be no-one's fault but ours, and if the consultants want to come back and laugh at our attempts, they are welcome to do so.
There is a possibility that I may come across as arrogant - I take pride in my work, I like to think that I am damn good at my job, and I can point to the fact that the IT in the company is light years ahead of what it was when I first joined them. I've had a number of people visit from outside, and they are generally very impressed with what we have achieved - we are way ahead of the curve for a business of our size. But I don't believe that I have the answer to every single problem, and I am willing to listen to ideas or advice from anyone.
Hopefully, this blog will be a useful resource for some people - if just one person finds something in my writing that helps their project in some way, I feel that it will have been worth it all.
Monday 12 March 2012
Still plowing on...
I've been tied up with work again over the last few weeks. Sometimes, it seems that the more we do, the less we have achieved.
We are still having problems with our first overseas site. It appears that there are still items of data that have not yet been loaded. Some of the information for foreign language descriptions of products already in the system, some of the purchasing information records, contact data and even some banking data amongst others. Note that I did ask the question more than once and was told each time that all of the data load had taken place.
I should also highlight that none of this data load has been tested in the Development or Test systems - they gave up doing that a couple of months before New Year, and the data load is being done directly into the Production system. Of course, it will do a couple of items, then fail because something is wrong with the actual data (which I accept is down to us) - but then they just correct it and carry on as if this is the most normal thing in the world. When I suggested that it might be better to at least try in the test system, I was told that was a waste of time.
As for data in the Test and Dev systems - It's been agreed that we are going to sort that out by doing a client copy. The last time that was done, it took more than a week to complete and we had all sorts of issues for a couple of weeks after. I hope that we don't get the same this time as I am trying to do a number of other projects and my staff and I will not have time to fix any issues.
Anyway, a couple of weeks ago, we had the guys from one of the other overseas sites visit with us, so that they could see how the SAP system works. I was told that we would see their whole project team, but only 3 people had travelled over. I've since had a chance to speak to the VP for the site, and made some points to him about the need for a full project team - I also put together some descriptions of work required which have been sent to him.
I've been asked to go over to them next week to talk to their people in a bit more depth. It turns out that they have been promised that our project team will all be available to help out, which is going to be a shock to most of those guys, as no-one has discussed this with them. I've received an invite to a meeting in April and I think they will all be told then - I've been asked not to discuss it with them yet.
There is also another issue that is causing a lot of anger. When we first started, the consultants set up a process within the Sales area that did not work the way we wanted - they told us that was how it had to be and was considered "best practice". How we wanted it to work was just not possible. Once it was working, they found some problems and we had to get a couple of people in to fix those.
When the new set of consultants were working on our overseas site, they set things up so that the process worked the way we wanted - and they said that was actually the way they normally do it. They couldn't change the process for us at that time, but have now agreed a date to do some work to re-do some things so that our Sales people can use the same process in the same way.
To re-cap - we paid for about 4 weeks to set it up, then another 2-3 weeks to "fix" it, and now we will have to pay for another 4 weeks to change it to work the way we wanted in the first place. I wanted to challenge them over this, but I've been told that we will just have to suck it up and pay the money.
The best thing - I've recently had an email from the SI asking if I would be prepared to recommend them! They want to use our site to show a "highly successful" implementation, and possibly even use us as a case study. Astonishing!
We are still having problems with our first overseas site. It appears that there are still items of data that have not yet been loaded. Some of the information for foreign language descriptions of products already in the system, some of the purchasing information records, contact data and even some banking data amongst others. Note that I did ask the question more than once and was told each time that all of the data load had taken place.
I should also highlight that none of this data load has been tested in the Development or Test systems - they gave up doing that a couple of months before New Year, and the data load is being done directly into the Production system. Of course, it will do a couple of items, then fail because something is wrong with the actual data (which I accept is down to us) - but then they just correct it and carry on as if this is the most normal thing in the world. When I suggested that it might be better to at least try in the test system, I was told that was a waste of time.
As for data in the Test and Dev systems - It's been agreed that we are going to sort that out by doing a client copy. The last time that was done, it took more than a week to complete and we had all sorts of issues for a couple of weeks after. I hope that we don't get the same this time as I am trying to do a number of other projects and my staff and I will not have time to fix any issues.
Anyway, a couple of weeks ago, we had the guys from one of the other overseas sites visit with us, so that they could see how the SAP system works. I was told that we would see their whole project team, but only 3 people had travelled over. I've since had a chance to speak to the VP for the site, and made some points to him about the need for a full project team - I also put together some descriptions of work required which have been sent to him.
I've been asked to go over to them next week to talk to their people in a bit more depth. It turns out that they have been promised that our project team will all be available to help out, which is going to be a shock to most of those guys, as no-one has discussed this with them. I've received an invite to a meeting in April and I think they will all be told then - I've been asked not to discuss it with them yet.
There is also another issue that is causing a lot of anger. When we first started, the consultants set up a process within the Sales area that did not work the way we wanted - they told us that was how it had to be and was considered "best practice". How we wanted it to work was just not possible. Once it was working, they found some problems and we had to get a couple of people in to fix those.
When the new set of consultants were working on our overseas site, they set things up so that the process worked the way we wanted - and they said that was actually the way they normally do it. They couldn't change the process for us at that time, but have now agreed a date to do some work to re-do some things so that our Sales people can use the same process in the same way.
To re-cap - we paid for about 4 weeks to set it up, then another 2-3 weeks to "fix" it, and now we will have to pay for another 4 weeks to change it to work the way we wanted in the first place. I wanted to challenge them over this, but I've been told that we will just have to suck it up and pay the money.
The best thing - I've recently had an email from the SI asking if I would be prepared to recommend them! They want to use our site to show a "highly successful" implementation, and possibly even use us as a case study. Astonishing!
Subscribe to:
Posts (Atom)