Lock and load

There is one thing about SAP that I do like - the data migration workbench. This allows you to load data from older systems. In the past, many other systems that I have worked with only allowed data to be loaded using SQL script - most just don't provide any support for this at all.

The actual process sounds a bit complex - well it's not that simple, but once you have done it a few times, it doesn't seem too bad. I would suggest that for most people, the hardest part is getting a project structure to manage it in place. Certainly, we started using a structure suggested by the consultants, but quickly found that it didn't meet our needs. Primarily because of the need to get the data loaded in the correct sequence, but also because we found some of the data was insufficiently prepared. We also found that they badly underestimated how it long it would take, I think mostly because they just didn't understand how much data we needed to move, even though we had actually explained this to them on several occasions.

Essentially, the data is extracted from the legacy systems in the form of flat text or comma separated files - these are then opened up in a spreadsheet. The records can be tidied up and the record attributes are placed in the colum headers (color, material number, weight, customer name, whatever). The data is then saved as a flat text file with the headers, and then the workbench allows this data to be processed record by record. Think of it in terms of a vb script, SQL script or even a simple Macro.

The problem of course is that the data has to be absolutely correct - if it isn't, the process will fail. This could be caused by the wrong data in a field, the wrong header or just simple mismatches. There is a process to re-run the failed import, but it still requires the data to be correct.

In many ways, this was the hardest part - people simply don't understand that a computer will only do what it is told. In many cases, they don't even see the error - many studies have been done that prove people often see what they want to see rather than what is in front of them. Several of the files had to be re-done, not just once, but several times. As a result, the load process for just one system, took well over 6 months, where we were told it would take a couple of months at the most.

However, once the structure is set up and proven, the good thing is that it can also be used on the other systems. This saves quite a bit of time, as it means you don't have to keep re-designing the process.

One thing we did find, was that it is better done by one person - we tried with a couple of us working on items in parallel. It just caused too many problems, and it was decided that it was wasting time. Since it was given to just one person, many of the problems we had had have faded. It also allowed a much more managed approach - as a result, the second half of the process became a lot easier.

Well enough for tonight - early start tomorrow.