A Web Service to transform Excel files into XML files

Rationale and Context

A SME trading company uses Talend solutions to transform Excel files into XML files.
This transformation was found to be the most simple solution to ensure the different departments of the company could easily work together. Indeed, if the use of Excel files is widely spread, some persons also need to work with a specific application, which works with XML files and a MySQL database. A Talend job was created to provide conversion and filtering means. This job was installed by the technical staff on all the machines, as a disguised batch script.


The company being in expansion, it was decided it would be more simple to expose the job through a single access point, rather than installing the job on each machine.

Contributors
No contributors found for: authors on selected page(s)

Solution

Overall solution

The retained solution was to expose it through a web application, that would call the job exposed a web service. The interest of the web service resides in the fact that add-ons and load-balancing facilities could be added later on, without redeveloping everything. Associated to the Talend target, these elements led to choose Petals ESB.

Thus, Excel files will be uploaded to the web application, which will then send them as attachment to the job.
The response will contain the resulting XML file as an attachment. The web application will then provide a link to download this XML file.

Petals Settings and Message Flows

Deployed configurations:

  1. One configuration for the Petals-SE-Talend, which embeds the job to execute. It exposes the job as a service inside Petals ESB.
  2. One configuration for the Petals-BC-SOAP component. It will expose the job as a Web Service outside Petals ESB.

Message flows:

If the web service is called from outside the bus, here are the different messages going through the bus.

  1. From Petals-BC-SOAP to Petals-SE-Talend: the input message.
  2. From Petals-SE-Talend to Petals-BC-SOAP: the response.

Configuration for the Petals-SE-Talend component

Creating the job

The job is made up of four Talend components:

  • The tFileInputExcel component loads the Excel file.
  • The tMySqlInput component retriives data from a MySQL database.
  • The tMap component filters the data contained in the Excel file by using database fields. It also performs some esthetic corrections.
  • Eventually, the tFileOutputXML component serializes the result as an XML file.


Here is what the job looks like.


Both kinds of files, as well as the database, deals with customers data.
Here are the different schemas, starting with the Excel one.


Here is the schema of the MySQL database.


And here is the schema of the output XML.

Roughly, the differences between the input and the output consist in merging the name and only keeping the clients whose revenue are over 50 K€.


The main configuration points for the tFileInputExcel and tFileOutputXML rely on the fact this job will take attachment files in input and return attached files in output.
This why the location of the XML and Excel files must be defined in two different context variables (respectively inLocation, outLocation).
Besides, outLocation will be have to be fixed by the client (the web application). This will be mentioned again during the export.


The configuration for the tMySsqlInput is not given here, since it is very standard.
And here is a screen shot of the tMap editor, showing the mapping of the inputs to the output.

Exporting the job for Petals

The job is exported as a singleton job. The reason is that the output location will be the same on every call.
In the scope of possible extensions, the Petals end-point will be generated by Petals on deployment. Thus, to deploy a new instance of this job inside Petals, there is no need to edit the or reexport the job. You can use the same deployment archive and install it directly.

Also, export the context variable inLocation as an In-Attachment.
And export the context variable outLocation as a Parameter and Out-Attachment.

Here is the export dialog.

Configuration for the Petals-BC-SOAP component

The creation of a SOAP configuration is generally fairly simple using the Petals Studio.
The goal here is to create a configuration for the Petals SOAP component, that will consume the Talend service.
It means every call this configuration will handle will be forwarded to the consumed service.

Please, make sure to use at least the version 4.0.3 of the Petals-BC-SOAP.
Previous versions did not manage MTOM attachments (in input and in output) correctly.


Since we expect a response, we will call the service with the InOut pattern.
Looking at the WSDL, we can also see that only the operation executJob is associated with this MEP.
Therefore, it will be the called operation.


Eventually, since we let the end-point be generated by Petals, we do not have to specify an end-point for the service to consume.
In fact, the interface name will be enough. Petals will decide which end-point must be called for this interface.


Here is the jbi.xml for the configuration.

<?xml version="1.0" encoding="UTF-8"?>
<!--
	JBI descriptor for the Petals' "petals-bc-soap" component (SOAP).
	Originally created for the version 4.0 of the component.
 -->
<jbi:jbi version="1.0"
	xmlns:interfaceNs="http://petals.ow2.org/talend/"
	xmlns:jbi="http://java.sun.com/xml/ns/jbi"
	xmlns:petalsCDK="http://petals.ow2.org/components/extensions/version-5"
	xmlns:soap="http://petals.ow2.org/components/soap/version-4"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	<!-- Import a Service into Petals or Expose a Petals Service => use a BC. -->
	<jbi:services binding-component="true">

		<!-- Expose a Petals Service => consumes a Service. -->
		<jbi:consumes
			interface-name="interfaceNs:Customers_3ServicePortType">

			<!-- CDK specific elements -->
			<petalsCDK:timeout>30000</petalsCDK:timeout>
			<petalsCDK:operation xmlns:op="http://petals.ow2.org/talend/">op:executeJob</petalsCDK:operation>
			<petalsCDK:mep>InOut</petalsCDK:mep>

			<!-- Component specific elements -->
			<soap:address>Customers</soap:address>
			<soap:remove-root>false</soap:remove-root>
			<soap:mode>SOAP</soap:mode>
			<soap:rest-add-namespace-prefix>soapbc</soap:rest-add-namespace-prefix>
			<soap:enable-http-transport>true</soap:enable-http-transport>
			<soap:enable-jms-transport>true</soap:enable-jms-transport>
		</jbi:consumes>
	</jbi:services>
</jbi:jbi>

Running the use case

To test this use case, you need to deploy the Petals-SE-Talend and Petals-BC-SOAP components in Petals.
Then, deploy the configurations for these two components.
Eventually, create a Java client, e.g. using Apache CXF or Axis 2. There are several such examples in the simple use cases.

Labels

uc-talend uc-talend Delete
uc-soap uc-soap Delete
petals petals Delete
tutorial tutorial Delete
se se Delete
bc bc Delete
soap soap Delete
talend talend Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.