Arejae.Com

I have been working on Data Warehousing in the Insurance Company for quite some time. Since 2001 to be exact. ETL is part of data warehouse component. My Introduction post about ETL is somewhere in 2006. Since then, so many things happened. I resign and come back again to the same company,doing the same thing. I’m not sure whether it is good or not but I enjoy doing my current job.

For my dear readers which is new with ETL,

ETL stands for extract, transform and load, the processes that enable companies to move data from multiple sources, reformat and cleanse it, and load it into another database, a data mart or a data warehouse for analysis, or on another operational system to support a business process.

Monitoring almost a thousand ETL job daily is not an easy job. Considering that a few problem might occured while running the jobs. Networking problem,source file not in proper format,database server down,database full,your SQL query getting so damn slow and many other things might occured.

Without proper ETL design, those problem might impact your SLA for example, your report should be ready at 9AM but due to those errors you cannot make it and re-run your ETL job will take quite some time if it’s not in a proper design.

So when you want to design your ETL job, you also need to consider what will happen if your program failed due to whatever reason. How you want to re-run the job ? Do you need to truncate your target table before re-run the job ? Do you want to consider automate the re-run task if the program failed due to network problem ? Do you need to handle any extra variables/parameter to your ETL job ? Do your ETL job need to re-run in the same day else the problem will getting bigger ? and…the list goes on.

I’ve no idea what to blog regarding ETL things. I also,haven’t found yet any other ETL programmer in Malaysia that blogged and I write this post simply because I don’t have an idea on what I’m going to blog.ehehe…

Well,if you are in ETL,Data Mining or any Data Warehouse related. Feel free to share your thought.

Till then.

Adios.

If you're new here, you may want to subscribe to my RSS feed or get my latest post directly in your mailbox. Thanks for visiting !