Unix Deployment and Scheduling of Talend Jobs

In this tutorial, we’ll go outside of the studio environment to show you how to set up your unix machine to run tasks outside of Talend Open Studio. We’ll create a simple task and then run it from a command line, then set a recurring task to do it daily.

Step 1: Create Job In Talend

To start off, simply add a tJava component into the canvas. This will be the only component in our job.

Secondly set up a local context variable in the context tab named “input”. Switch to “values as a table” and make the Default setting “standard”.

Lastly insert the following code into the tJava component. With the display code to check the value of context parameter input.

Save when everything is done.

 

Step 2: Export to Folder

Right click the job name in your repository and select “Build Job”.

 

Choose the location you want a zip file to be sent to. If you don’t have to move it anywhere else, select the “Extract the zip file” option. This will save the item and allow it to be runnable in the location specified. The “All” option downloads both a batch file for Windows and Shell file for Unix.

Step 3: Run on Command Line

In this demo we’ll be working with unix. Remember .bat is for windows and .sh is for unix.

Navigate to the location of the sh file in a command prompt and enter the command “bash <jobName>_run.sh”. The job shown here is simpledeploy, thus simpledeploy_run.sh. Note the text “Standard was called!”.

To change the context variable at run time, add it after the original command so:

“bash <jobName>_run.sh –context_param input=<not standard>”. Any value that is not “standard” will result in the text “Manual was called!”

Step 4: Schedule a Recurring Task

Many people will want to schedule their talend jobs as recurring tasks to run every day, week, etc. In unix, this can be done with a cron job.

To open up the list of cron jobs, type “crontab -e”.

Here you can schedule tasks on each line with the following format:

1 2 3 4 5 /path/to/command arg1 arg2

Each number means:

1: Minutes, 2: Hours, 3: Day, 4: Month, 5: Day of the Week

The following entry runs every day at 11:00 PM:

Save and quit to have this job run forever.

 

  1. sonu Reply

    Hello ,
    I am new to Talend DI .I want to read Unix flat file from xyz.com unix server with path /prod/home01/data/serial/input.dat. I can create schema in tFileInputDelimited component if required.
    But I dont know how to read file directly with tFileInputDelimited comp. from xyz server and process in my Job ?

    Thanks in Advance !!!

    • admin Reply

      Hi Sonu. You will first need to download the file to the local machine. You can use something like tFTPGet to do that.

      TFileInputDelimited assumes the file is on your local machine, not a remote server.

  2. sonu Reply

    Thanks Admin it helped !! I am able to connect to remote server. I am reading CSV but its failing with below error

    [statistics] connecting to socket on port 3903
    [statistics] connected
    Exception in thread “main” java.lang.Error: Unresolved compilation problems:
    msg_tFTPGet_2 cannot be resolved
    Invalid escape sequence (valid ones are \b \t \n \f \r \” \’ \\ )

    at firstproject.madmod_ld_0_1.Madmod_LD.tFTPGet_2Process(Madmod_LD.java:413)
    at firstproject.madmod_ld_0_1.Madmod_LD.runJobInTOS(Madmod_LD.java:806)
    at firstproject.madmod_ld_0_1.Madmod_LD.main(Madmod_LD.java:663)

Leave a Reply

*

captcha *