Solution

Overall solution

Each shop owns a Petals node, as well as the collecting central.
Each shop node has installed the Petals-BC-FileTransfer component to monitor a drop-in directory.
The collecting central node has installed the Petals-SE-Talend component. This one is in charge of loading the CSV files it receives and put their data into a database.

Thus, every time a CSV file is copied into the drop-in directory of a shop, this CSV files will be sent as an attachment to a Talend job to be processed.

Petals Settings and Message Flows

Deployed configurations:

1. On each shop node, there is an instance of the Petals-BC-FileTransfer component, with one configuration to load CSV files and send them to the job.
2. On the collecting central, there is one configuration for the Petals-SE-Talend, which embeds the job that will process the CSV files.

Message flows:

1. From Petals-BC-FileTransfer to Petals-SE-Talend

Notice that one node can emit several successive messages, if several files were dropped at once.
Besides, several nodes can send messages at the same time. Therefore, several instances of the job may have to be executed concurrently.

Configuration for the Petals-SE-Talend component

Creating the job

The job is made up of two Talend components:

• The tFileInputDelimited component is in charge of loading a CSV file.
• The tMySqlOutput component is in charge of writing the sent data into a MySQL database.

Here is what the job looks like.

The schema of the CSV files is the same for every shop, and for the database.
It is shown on the following screen shot.

The properties of the tMySqlOutput are not described here, since there are very common.
And the only specific property of the tFileInputDelimited is that the location of the file to load is defined by a String context variable, called inLocation.
Its value will be set by Petals when the job is instantiated.

Exporting the job for Petals

The job is exported as a non-singleton job.
It is very important to uncheck the singleton option during the export. Otherwise, concurrent execution will be impossible and messages will be rejected (meaning that some CSV files will not be processed).
And the inLocation context variable must be exported as as In-Attachment.

Configuration for the Petals-BC-FileTransfer component

This component is in charge of scrutinize a drop-in directory and detect every CSV file that would be moved into it.
When a CSV file is found, it is loaded and sent as an attachment to the processing Talend job.

Please, make sure to use at least the version 3.0-SNAPSHOT of the Petals-BC-FileTransfer. Previous versions did not support the definition of a message skeleton (so that a valid message is sent to the target service, respecting the WSDL schemas).

The directory to monitor is the drop-in directory (that is here defined using an environment variable called DropInDirectory).
This directory will be polled every five seconds.
And the files to check are CSV files, i.e. files having a *.csv extension.

The service to consume is the Talend service that was exported above.
The Message Exchange Pattern must be InOnly (no other MEP is allowed for this component in consume mode).
The only operation that supports this MEP in the WSDL of the Talend job, is executeJobOnly.
Eventually, the skeleton of the message to send can be generated with a tool like SoapUI.
In this case, the sent message will only contain a reference to the attachment.

Once exported, this configuration can be used for all the distributed nodes.
Here is the skeleton of the message to send to the Talend service.

<tal:executeJob xmlns:tal="http://petals.ow2.org/talend/">
	<tal:contexts />
	<tal:in-attachments>
            <tal:fileLocation>
            	<tal:fileContent>$attachment</tal:fileContent>
            </tal:fileLocation>
	</tal:in-attachments>
</tal:executeJob>

And here is the jbi.xml of the FileTransfer configuration.

<?xml version="1.0" encoding="UTF-8"?>
<!--
	JBI descriptor for the Petals' "petals-bc-filetransfer" component (FileTransfer).
	Originally created for the version 2.5-snapshot of the component.
 -->
<jbi:jbi version="1.0"
	xmlns:filetransfer="http://petals.ow2.org/components/filetransfer/version-2"
	xmlns:gen0="http://petals.ow2.org/talend/"
	xmlns:generatedNs="http://petals.ow2.org/talend/"
	xmlns:jbi="http://java.sun.com/xml/ns/jbi"
	xmlns:petalsCDK="http://petals.ow2.org/components/extensions/version-5"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	<!-- Import a Service into Petals or Expose a Petals Service => use a BC. -->
	<jbi:services binding-component="true">

		<!-- Expose a Petals Service => consumes a Service. -->
		<jbi:consumes
			interface-name="generatedNs:CommandsCollectorServicePortType"
			service-name="generatedNs:CommandsCollectorService"
			endpoint-name="CommandsCollectorEndpoint">

			<!-- CDK specific elements -->
			<petalsCDK:timeout>30000</petalsCDK:timeout>
			<petalsCDK:operation>gen0:executeJobOnly</petalsCDK:operation>
			<petalsCDK:mep>InOnly</petalsCDK:mep>

			<!-- Component specific elements -->
			<filetransfer:read-directory>${DropInDirectory}</filetransfer:read-directory>
			<filetransfer:transfer-mode>content</filetransfer:transfer-mode>
			<filetransfer:file-pattern>*.csv</filetransfer:file-pattern>
			<filetransfer:polling-period>5000</filetransfer:polling-period>
			<filetransfer:base-message><![CDATA[<tal:executeJob xmlns:tal="http://petals.ow2.org/talend/">
	<tal:contexts />
	<tal:in-attachments>
            <tal:fileLocation>
            	<tal:fileContent>$attachment</tal:fileContent>
            </tal:fileLocation>
	</tal:in-attachments>
</tal:executeJob>]]></filetransfer:base-message>
		</jbi:consumes>
	</jbi:services>
</jbi:jbi>

Running the use case

To test this use case, install the Petals-SE-Talend and the Petals-BC-FileTransfer components on the target nodes.
Define the environment variable DropInDirectory.
Then, deploy the associated configurations in Petals.

To run the test, simply drop a CSV file in the drop-in directory and check the database then.