Creating and Importing Files on Sources

After completing the source definitions and testing them successfully, you can start creating database tables and/or filesystem files to transfer to hadoop. You can create multiple database tables and files per source and transfer them seperately by defining jobs. Remember that, every transfer is asynchronous and will be running as a background process. Once the transfer ends successfully the status of the file will be updated as FINISHED, while the status will be RUNNING during the transfer.

To login use browser and insert the portal address. By default portal works on the port 8080 of the xenn.lake installed server.

http://<hostname>:8080/xennlake_web

Go to Integration Module

Click on the <INTEGRATION> Link on top of the web page. This will lead you to the integration module.

Click <PROPERTIES> button for the source details

This will lead you to the details page of the defined connection. If your connection is the one type of a supported databases, then file represents the database table. If your connection is an operating system file system, then the file represents the text file you want to import to Hadoop. Or if your connection type is flumeAvro, then the file represents the flume sink.

Click on the <CREATE NEW FILE> button to open the new file popup window and supply the necessary information.

Insert Necessary Information

From the <New File> popup window;

File Name: The name of the file to be transferred to Hadoop file system. For database sources, the file name is the name of the table. For file system sources, the file name is name of the file to be transferred to Hadoop.

Incremental Column: If the source type is a database and the database has a column value which increments by every insert then xenn.lake can use this value to incrementally transfer the data to Hadoop.

Incremental Value: The starting value of the database table column where the transfer starts. Every time the transfer completes this value is updated in the repository so the next transfer will start from this value to transfer the data incrementally.

Click the <SAVE FILE> button to save the file definition.

Start the File Transfer

After saving the file. New created file will be listed in the <List of Files> table. On the right side of the listed file (Actions Menu) click on the <START TRANSFER> button.