Data Load Utility

DataLoad utility is the new enhanced business object based utility available in websphere commerce where input data is converted to business objects and then business objects to physical data.

Following user Roles generally interact with dataload utility in Websphere Commerce:

1. Business user- Responsible for preparing business data / input data.
2. Developer - Responsible for creating data source templates (EX: CSV files.. From FEP6 onwards XML input files are also accepted) business object mappings in business object configuration files (Ex: XML files) and customizing the data load utility.
3. Site Administrator - Responsible for day- to - day dataload utility. - Like creating environment based files, Run the coniguration files to load the data into websphere commerce database.

There are 3 configuration files involved in dataload utility
1. Environment configuration file. - prepared by Site administrator
    <_config:DataLoadEnvConfiguration >

                 ........

    </_config:DataLoadEnvConfiguration>
This configuration file has -  dbname, dbusername, password, server, port number, schema information.
Ex;
<_config:Database type="db2" name="XXX" user="db2admin" password="Szv3RSJ0AYLgRmGpKekdvw==" server="localhost" port="50000" schema="db2admin" /> Also has information about storeIdentifier like
<_config:BusinessContext storeIdentifier="Extended sites catalog asset store" catalogIdentifier="Extended sites catalog asset store" languageId="-1" currency="USD"/>

2. DataLoad configuration file. - prepared by Site administrator
   <_config:DataLoadConfiguration >
              .....
   </_config:DataLoadConfiguration>
 This file has the entry for the data input file, business object configuration file, environment configuration file.
EX:
<_config:DataLoadEnvironment configFile="wc-dataload-env.xml" />
<_config:LoadOrder commitCount="1000"  maxError="10" batchSize="500" dataLoadMode="Insert"><_config:property name="firstLineIsHeader" value="true" />
 <!-- Store Locator -->       
 <_config:LoadItem name="StoreLocation" businessObjectConfigFile="wc-loader-store-location.xml" >
              <_config:DataSourceLocation location="myData.csv" />
 </_config:LoadItem>

3. Business object configuration file. - prepared by Developer
   <_config:DataloadBusinessObjectConfiguration
            ...........
   </_config:DataloadBusinessObjectConfiguration>
This files contains entry for :
OOB CSV Reader
<_config:DataReader className="com.ibm.commerce.foundation.dataload.datareader.CSVReader" firstLineIsHeader="true" >

 <_config:BusinessObjectBuilder className="com.ibm.commerce.foundation.dataload.businessobjectbuilder.TableObjectBuilder" >

Data Load utility - user interaction diagram 

Source - Infocenter

1. The business user provides the developer with the business data.
2. The developer creates a data source template, which defines how source data must be formatted before the data is loaded.
3. The developer also creates the business object configuration file, which defines how the Data Load utility maps the input data to the business object and how to transform the business object to physical data.
4. The site administrator uses the business object configuration file to define and create the load order configuration file.
5. The site administrator sets the store and database settings in the environment configuration file.
6. The business data is formatted according to the rules of the data source template before the data is loaded to the database.
7. The formatted source data is provided to the site administrator.
8. The site administrator runs the Data Load utility along with the three configuration files (environment, load order, and business object configuration files) to load the formatted source data into the WebSphere Commerce database. After the utility runs, the site administrator also verifies the results of the load.
9. The business data is available in WebSphere Commerce to be managed by the business user.

How to Run dataload utility for custom tables:
1. Take back up of tables that are going to be modified / take DB backup.
2. Note the count of rows for the tables that are going to be modified.
3. Generate a CSV file for the input data - verify the CSV file using CSV reader just to make sure that the CSV file is error free.
4. Create a new folder with the name of unit for which you want to load data.
 For ex: F:\IBM\WebSphere\CommerceServer70\samples\DataLoad\StoreLocactor
5. Copy the wc-dataload-env.xml, wc-dataload.xml and wc-loader.xml files from different folders that you already have into F:\IBM\WebSphere\CommerceServer70\samples\DataLoad\StoreLocactor and modify the required fields in these xml files.
6. Name the xml files to avoid confusion like Ex:
  env file as - wc-dataload-env.xml
  loader file - wc-dataload-stloc.xml (This is the entry point for execution)
 Business Object Configuration file - wc-loader-store-location.xml
6. Check the wc-dataload-env.xml file for DB configurations.
7. Update the proper Input CSV file path, business object config file path in wc-dataload-stloc.xml.
8.Add the table column mapping with the CSV column in wc-loader-store-location.xml.
9. Run the dataload command from command prompt.

F:\IBM\WebSphere\CommerceServer70\bin>dataload F:\IBM\WebSphere\CommerceServer70\samples\DataLoad\StoreLocator\wc-dataload-stloc.xml

Useful Link : https://www.youtube.com/watch?v=jCVOwqH0Rhw
http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.data.doc/refs/rmldataloadsample.htm