Data Upload
Recently one of the projects I was architecting went live. All the data that was stored in an excel sheet prior to this project was be ported to a DB. Data was loaded to lower environment multiple times and the real data was loaded to Staging Database prior to production. The story is good so far.
On the day of production, the same data was copied from Excel and uploaded to production with help of DBA and his scripts. When the data was verified next day, to my horror, we discover couple of rows with junk characters at the end of primary key. The excel sheet had couple of white spaces in the end for that column, it is a mystery how it got converted to special characters.
A production CR was created to delete the row, then a new set was to be added with additional rows that were not in production. The second round of data upload was to take place using ETL too. I was little at peace with this approach as the insert would happen only to the batch table and then it will moved to real table.
Once the upload to batch was done, production support team verified if the data was all good this time. The bad rows with characters in the end was good now, but couple of new rows had leading white spaces now. So much for the verification. Thankfully, it was not the real table, the bad rows were deleted with help of DBA and new rows with spaces trimmed were loaded again.
In an enterprise you would expect, QA team would have checked uploading with white spaces on the primary in their testing phase and reported it and same with the DBA and ETL, they should have added trim functionality on primary keys before uploading. Aren't the basics that need to be followed. Sometimes, developers and QA testers are majoring in the majors and dropping the ball when it comes to testing the minors.
In my next phase we are going to have a whole new UI to handle this data, so we won't need ETL anymore. Hopefully group developing the UI do a better job here. I am sure, will discover new errors in production.
On the day of production, the same data was copied from Excel and uploaded to production with help of DBA and his scripts. When the data was verified next day, to my horror, we discover couple of rows with junk characters at the end of primary key. The excel sheet had couple of white spaces in the end for that column, it is a mystery how it got converted to special characters.
A production CR was created to delete the row, then a new set was to be added with additional rows that were not in production. The second round of data upload was to take place using ETL too. I was little at peace with this approach as the insert would happen only to the batch table and then it will moved to real table.
Once the upload to batch was done, production support team verified if the data was all good this time. The bad rows with characters in the end was good now, but couple of new rows had leading white spaces now. So much for the verification. Thankfully, it was not the real table, the bad rows were deleted with help of DBA and new rows with spaces trimmed were loaded again.
In an enterprise you would expect, QA team would have checked uploading with white spaces on the primary in their testing phase and reported it and same with the DBA and ETL, they should have added trim functionality on primary keys before uploading. Aren't the basics that need to be followed. Sometimes, developers and QA testers are majoring in the majors and dropping the ball when it comes to testing the minors.
In my next phase we are going to have a whole new UI to handle this data, so we won't need ETL anymore. Hopefully group developing the UI do a better job here. I am sure, will discover new errors in production.
Comments
Post a Comment