Rationale and Context
The application XXX is a legacy application that daily produces various kinds of logs.
Each log is a positional file, where information is ordered by column of a determined length.
Due to recent hardware failures, it was decided that some of these logs should be backed up in a more robust way.
In addition, a new application upgrade is planned, which consists in using a new monitoring console with this application.
Eventually, the Information System Manager wants to define and set up additional processes, implemented with BPEL. These new processes will have to interact with the existing application.
Some of these interactions may involve log backups in case of emergency situations.
Hence, for a long-term storage, and in order to make the application events visible in this new monitoring console, it was decided the main information from these logs would have to be archived every 3 days in a data center. This data center will be accessed by the monitoring console to display the events the user queries. The log storage will also act as a safety measure in case where other hardware failures would occur.
Solution
Overall solution
The logs processing was decided to be handled with Talend Open Studio, while the BPEL processes will have to run inside Petals ESB.
The requirements naturally led to a native integration between both products.
Besides, it was proposed that the execution schedule would be defined in Petals.
Thus, with this solution, the logs are copied in a temporary directory, loaded, filtered and written to a database through a Talend job.
The loaded logs are then moved into a backup directory. This way, one log file is processed only once before being archived.
This Talend job is exposed as a service into Petals, using the Petals-SE-Talend component. This way, it can be called by any Petals service, including a BPEL process.
And it can also be scheduled to be executed on a regular basis, every day in this case.
The execution schedule is handled by the Petals-SE-Quartz component.
Every 3 days, at the same time, it will send a message to the Talend job, so that the log transfer is made.
 | This use case only describes the way the logs are saved periodically. |
Petals Settings and Message Flows
Deployed configurations:
- One configuration for the Petals-SE-Quartz component. It will trigger message sending to execute the job.
- One configuration for the Petals-SE-Talend, which embeds the job to execute.
Message flows:
- From Petals-SE-Quartz to Petals-SE-Talend
Configuration for the Petals-SE-Talend component
Creating the job
The job is made up of six Talend components:
- The first sub-job copies all the log files into a temporary directory.
- The tFileList components gets all the *.log files from the directory System.getProperty( "user.home" ) + "/logs"
- The log file is copied into a temporary directory using a tFileCopy component. The original file is deleted.
- If the first sub-job worked fine, the second sub-job is run.
- A second tFileList components gets all the logs file from the temporary directory.
- For every found log file (iterate connection), the tFileInputPositional component loads its content.
- The tMysqlOutput component receives the content from a log file and sends it into the table logs from the logstorage database.
- Eventually, a second tFileCopy is in charge of copying the log into the backup directory. The original file is deleted.
Here is what the job looks like.
Here are the properties of the first tFileList component.
Here are the properties of the first tFileCopy component.
Here are the properties of the second tFileList component.
Here are the properties of the tFileInputPositional component.
Here are the properties of the tMysqlOutput component.
Here are the properties of the second tFileCopy component.
Eventually, here is the schema of the tMySqlOutput and tFileInputPositional components.
Exporting the job for Petals
The selected export options are the default ones (singleton job, no generated end-point).
Configuration for the Petals-SE-Quartz component
Exposing the Talend job as a service inside Petals allows it to be called by any client.
This features allows the use of the Petals-SE-Quartz component.
Every 3 days, this component will be in charge of sending a message to trigger the execution of the BackupLogs job.
The job will be called every three days, let's say at noon. It corresponds to the following CRON expression.
And the XML message to send can be found, as an example, with SoapUI.
In this case, no parameter is required.
<tal:executeJob>
<tal:contexts/>
<tal:in-attachments/>
</tal:executeJob>
The configuration below was generated with the Petals Studio for the version 1.1 of the Petals-SE-Quartz component.
<?xml version="1.0" encoding="UTF-8"?>
<!--
JBI descriptor for the Petals' "petals-se-quartz" component (Quartz).
Originally created for the version 1.1 of the component.
-->
<jbi:jbi version="1.0"
xmlns:generatedNs="http://petals.ow2.org/components/ftp/version-3"
xmlns:jbi="http://java.sun.com/xml/ns/jbi"
xmlns:petalsCDK="http://petals.ow2.org/components/extensions/version-5"
xmlns:quartz="http://petals.ow2.org/components/quartz/version-1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
use a BC. -->
<jbi:services binding-component="false">
consumes a Service. -->
<jbi:consumes
interface-name="itfNs:BackupLogsServicePortType"
service-name="srvNs:BackupLogsService"
endpoint-name="BackupLogsEndpoint"
xmlns:srvNs="http://petals.ow2.org/talend/"
xmlns:itfNs="http://petals.ow2.org/talend/">
<petalsCDK:timeout>30000</petalsCDK:timeout>
<petalsCDK:mep>InOnly</petalsCDK:mep>
<quartz:cron-expression>0 0 12 1/3 * ?</quartz:cron-expression>
<quartz:content><![CDATA[<tal:executeJob>
<tal:contexts/>
<tal:in-attachments/>
</tal:executeJob>]]></quartz:content>
</jbi:consumes>
</jbi:services>
</jbi:jbi>
Running the use case
To test this use case, the first valuable thing to do would be to change the CRON expression, so that the jobs is executed every minute instead of every day.
should work.
Then, add a log file in the directory the tFileList will list.
As an example, you can use the following log sample. It will be inserted in the database (let the job create the table).
Deploy the Petals-SE-Quartz and Petals-SE-Talend components in Petals, as well as the two service-units.
Wait for a minute and check the Petals console and the database.
The console should display information about the request processing in the Petals-SE-Talend component.
The database should see the logs table created and filled-in. And the log file should have been moved into the backup directory.
|
E-mail