New IT system – What about data migration?
Throughout my career I’ve been involved in projects which have covered: introduction of a new IT software solution, a new version of the same software or migration to a different vendor.
In all cases getting historic data (for clarity I use the term “data” to mean electronic information and documents, including paper hard copy) into the “new” system is a major consideration.
Whilst a lot of time has been spent selecting the new system…. the important element of migrating historic data into it appears to be an afterthought.
This article is an evolving collection of my own thoughts in very general terms as each situation can be different but, there are, I feel common threads…. regardless of whether the new system is a:
•content management system (CMS),
•document management system (DMS),
•customer relationship management (CRM),
•product data management (PDM),
•product lifecycle management (PLM),
•enterprise resource planning (ERP)
…. or any other IT system with a three letter acronym 🙂
1. What to import/transfer
Firstly a key stage, and this can save you a lot of time further down the line, decide – with involvement from your key stakeholders – what you want to import/transfer into your new system.
You may have encounter different opinions/views with some who want absolutely everything, every bit of data since year dot …but do you really need all?
Using the old adage of “rubbish in, rubbish out” think of this as a potential opportunity to cleanse your data… your “system is only as good as the data within in it” I’ve heard many times.. and it’s true.
Will only having “useful” data in your system allow you to get to information/respond quicker, make more accurate decisions, allow the actual system to perform faster?
Also try to understand and get to the root behind anyone insisting on all historic data being present in the new system.
Consider doing a MOSCOW (Must have, Should have, Could have or Won’t have this time) analysis of the different data you have and understand what is essential from day one and what can be added later, maybe as a phased data migration approach?
In the main I believe there are three main options of transfer; all data, no data or selected data.
a)Transfer all data. The “big bang” approach… having everything in.. from day one. This may be an easy option to pick and perceived as potentially having less risk (all data transferred, so nothing will be missed)… but if you are talking big volumes of data this can have many negatives.. including;the risk of inaccurate information being transferred, longer time and more overhead to perform the data preparation as well as transfer.. with potential knock-on effect of new system being delivered late(r).
b)Transfer no data. Decide not to transfer any historic data. Draw a line in the sand, pick a date and say that from this time forwards all information should be in this new system. Using knowledge that anything before this date.. will exist in a system elsewhere or maybe deleted/destroyed (if possible – data might need to be kept for a period of time for compliance so be careful of this).
c)Transfer selected data A hybrid of the two other options. Pulling out/filtering key historic data to go to the new system. (If so, decide what happens to the rest of the “redundant data” – do you need to retain for compliance purposes? can it be archived? etc).
The biggest challenge with this is how to identify what is the important “key” data and the resource that might be needed to pick/filter this out.
With transferring selected data the other option to consider is whether all of this data needs to be in the system from day one…
…if a staged approach where some of the historic “nice to have” data is gradually entered over time.
For example: Day one data could be just; current customers/accounts/projects, with related information included… then other data added…
on a “as and when needed”…
…. scheduled as an admin task (maybe weekly or monthly) to clear the historic data backlog to go into the system.
2. Current data format and volume
Before looking at your own data I would advocate reviewing what format is required by new system. Especially if the new system has an import/upload facility that you could use and you are dealing with large volumes of data to be transferred.
Look at things such as:
•the data fields (and their order) which are required by the new system
•if the fields have a certain format to be following and which fields are mandatory
•what file types will be accepted by the new system
•if there is a file size limit that can be uploaded to the new system
•plus any guidance from the new system vendor about the data importing process.
If the new system can be configured to your requirements (e.g. fields, file structure/ metadata categories, security/access rights, etc) it is important that this tailoring is decided first… as far is possible…before migration.
All of these considerations are important as it will indicate any preparation work of existing data needed, to get it into the format ready for importing.
(Caveat: If you are dealing with small volumes of historic data going into the new system then consider manual data entry/uploading as a potential option… as this may actually be quicker than manipulating/reformatting the data then importing it).
OK… so now you know the ideal format the data needs to be in for ease of importing… and what data you want to import.
Now it’s looking at what form your current data is in, where it is located and the volume.
Breaking that down….
….current format…. Is your data electronic already or in hard copy?
If electronic… what format is this in? how frequently is it used? What are the low/peak usage times (day of week/times)?
By gaining an understanding of the volume of data to be transferred it can give you a starting point* for calculating both the time to prepare/manipulate all existing data and also the actual transfer time
(*you would also need to know approximate time for preparing a standard piece of data and time it would take to be import, to do this multiplication – these times could be calculated by taking representative sample data – then timing the preparation and importing as a test into the new system).
3. Remember data security and access
When looking at current data it is vital that data security is not forgotten.
Does the data being selected to be imported already have any restrictions on it? with regards to who can access, view, edit, save, print, etc? If so make sure replicated/adhered to in new system.
After identifying which option you are going to take for data migration (a, b or c from section 1 of this piece) then do investigate and record the existing security and access applied to the data.
If the data is currently in hard copy, by virtue it is in a certain filing cabinet (for example) or draw may mean that access to only certain staff (who have the physical keys to that cabinet) needs to be reflected when the data is digitised and imported into the new system.
This might be an opportunity to define different security or access levels, maybe introduce a new scheme or make it more granular.
4. Preparation of data
The type of data preparation you do for transfer is dependent on which one of the three data migration approaches is chosen from what I mentioned in point 1
(a. transfer all data, b. transfer no data or c. transfer selected data).
If you are progressing with options a or c then preparation of the data can potentially be a larger piece of work than the data migration itself… and should be included in the overall data migration plan (covered later).
If the data is in paper format…consideration needs to be given how to digitise this.
The simple act of getting paper files ready for scanning… e.g,. removing staples, collating into a certain order, then the actual scanning… can all take time.
But for paper files it is important to consider that it is not just about scanning the paper but in some way also categorising this/making it meaningful e.g. the filename, applying metadata, saving to a certain folder in a structure, etc
I remember seeing one example in the past where a department in an NHS hospital had commissioned a bureau to scan all of their paper records to DVD… they did this… but unfortunately that’s all they did… so whilst all the paper records were now electronic… they had no meaning full categorisation, division or filename… just file1, file2, file3, etc on a DVD…. So that meant staff had to spend (waste) time opening each and every single file until they got to the information they wanted.
With electronic data the preparation can cover a range of activities from: mapping existing data fields to the new systems data fields, adding new field data (for mandatory fields in new system), editing data to match the format in a new system, etc.
As mentioned previously, but is worth repeating, this preparation can take time and it needs to be factored into your planning time lines as well as considering who does the work. Various options exist for who performs the preparation, from using automated tools, in-house staff (do they have time? Maybe use temporary staff?), or outsourcing e.g. scanning bureau. However in deciding on the “who” the element of categorising uncategorised information or selecting relevant data may need empowered staff with appropriate knowledge.
Careful timing of when the actual data preparation is performed should be evaluated when producing the data migration plan… and also thought given that if the data is being worked on it maybe unavailable to the current users. Where possible try to make the important data that’s being frequently used being taken “offline” for manipulation as close to the migration date as is possible – to limit unavailability/disruption to the business.
5. Create the overall data migration plan
In an overall data migration plan I would recommend this actually consisting of five “plans”: project plan, communication plan, risk management strategy, risk register and a “roll back” plan.
Going into a few of these…
A collection of all the tasks needed to be completed for the data migration,
(covering, not an exhaustive list: identification of historic data to be transferred, data preparation, progress reports/meetings, communication plans, test /trial run, full data migration, testing, decommissioning/archiving, etc) the dependencies between these activities, the time allocated to each of these tasks and who they are allocated to.. will give an overall time line and plan.
In this type of project I would always suggest having a sample of the typical/representative data to be migrated and doing test run of the preparation and migration – this will both prove the process and also give you an indication of timings. Alternatively on the “How long it will take?” you could use past experiences (of yourselves or the vendor).
The timing (day/time) of the “actual” migration during planning should consider; the users, times of peak and low use, parallel running of two systems.
Parallel running – if upgrading to new version of software, or changing vendor is it possible to keep running the old system right up until the point or after go live with the new system?
Once the actual migration of data is performed… that is not the end … and testing, verification and validation should be included within the implementation plan. This stage is to ensure that the migration, especially if using automated methods, worked and that data hasn’t been corrupted.
Once everything is up and running for a period of time then look to switch off the old system altogether.
Have the types of communication, and the content for these, planned and factored into your project plan. This could be a message to inform users when the existing system/data should not be used from, when the new system could be used and details who to contact if any issues spotted, or – in worst case scenario/roll back situation – informing users to continue work but with old system.
A list of all specific potential risks that could occur in the project covering items such as risk description, probability, potential impact, response, status, etc.
A few risk examples, again this isn’t an exhaustive list, that you might need to mitigate/plan against…
• Data not in correct format for importing/migration
• Data preparation taking longer than expected
• Data still migrating and encroaching into the start of next business day
• Data migration fails part way through or fails altogether
• Data loss/corruption
• Data being migrated isn’t the latest/ most up-to date
• Staff prevented access to site / building / room / system “outside of standard working hours” required to perform the migration
• Use of “Old system” should be stopped from x date but new system isn’t ready for use
Roll back plan
A detailed plan elaborating on actions to take if risk of data migration failed happened.
It is very important to have a “roll back” if things don’t go according to plan during the actual migration… how can we revert? or what can we do to ensure that people can perform their duties utilising the data if migration isn’t successful?
Elements you could include/cover:
• Perform a data back-up before any preparation work is done, then after the preparation (but before migration)
•Record any security and/or access rights
•Ensuring availability of old system that can revert to
As I said from the outset these are an evolving collection of my knowledge and experience and hope this is a useful input but do…do plan, plan, plan your data migration carefully… that includes covering; communication, risk management strategy with risk mitigation and “roll back”.
Have you encountered any of the stages/challenges in my article?
I welcome sharing thoughts and view points on this diverse subject.
Please also feel free to share any links in the comments box to other useful articles on this topic.
It would be great for this to be a central point reference to useful data migration tips, tricks and advice… to really help others.