Creating Streaming File Source

Login to xenn.lake Portal

To login use browser and insert the portal address. By default portal works on the port 8080 of the xenn.lake installed server.

http://<hostname>:8080/xenn.lake_web

Go to Integration Module

Click on the <INTEGRATION> Link on top of the web page. This will lead you to the integration module. By there are two system generated file system sources which cannot be deleted.

generalFileInput: Is for the custom operating system files which can be uploaded manually and let xenn.lake to import them to the Hadoop File system.

downloads: Are the predefined source definition for the uploaded files via the INTEGRATION module. You can directly upload your files from the UPLOAD button which is in the left side of the INTEGRATION module. This uploaded files are going to be stored in the downloads source definition.

Select <New Source>

From the left menu, click on the <NEW SOURCE> button to open the pop-up windows for the new connection.

Insert Necessary Information

From the pop-up window;

Source Name: Name of the defined source connection.

Source Type: Select the source type as flumeAvro which represents the flume.

IP Address: The IP address of the listener works on.

Port: The port number of the flume listener.

Click the <SAVE CONNECTION> button to save the connection.

Create the file for the flume integration

In order to test and run the streaming file source, first you need to define at least one file for this type of source. This file definition represents the sink, therefore you can have more than one file (sink) definition for every opened port. You can decide to have more than one file definition on every port, or every streaming file may have their own source definition on different port. This is by design consideration and depends on the customer architecture.

To create the file definition Click on the properties button which is on the right side of the created connection. This will lead you to the details of the connection.

Click on the <CREATE NEW FILE> button.

Insert the name of the file, consider this filename is the sink name of the flume.

You do not need to enter the Incremental Column; and Incremental Value parameters. These parameters will be used for the database tables for incremental data loading option.

Click on the <SAVE FILE> button to save the file information.

Start the File Transfer

After saving the file, new created file will be listed in the List of Files table. On the right side of the listed file (Actions Menu) click on the <START TRANSFER> button. The Status will be changed from the <DOWN> to <UP>. Now you can check the status of the connection.

Test Connection

After saving the connection and creating the file, go back to main Integration Page and click on the <TEST> button to test the flume connection.