Welcome To Support Community

Pipeline Pilot

Advanced Search
Ask Search:
Joe_BJoe_B 

SOAP connector result transfers

Hi,

I’m hoping someone may have seen this before and know the answer…

I’m seeing a strange thing when using the AEP SOAP connectors. Whenever I run a protocol I’ve written to call a pipeline pilot protocol via SOAP the longest part is the Start Job (upload data) and Get Results. The more results I have the longer it takes (as expected), however the time taken is exponential to the file size!

For example to return 10000 results generates a 1556Kb sdf file. This take over 3 minutes to return in Get results. However the actual SOAP service calculates the results in under 20 seconds (not dependant on file size)! So the transfer of data via SOAP is most of the time.

Uploading or downloading these files on the server from the client is pretty much instant, so it’s not a bandwidth / network issue as far as I can see.

The time it takes seems to increase exponentially relative to the file size for upload / download.  The table below shows doubling the data in / out of the SOAP service creates delays much longer than double the time.

Records

Start job (seconds)

return data (seconds)

File size (Kb) upload

File size (kb) download

1000

0.14

1.76

13

150

5000

1.56

37

66

750

10000

5.77

152

131

1500

20000

22.2

>600

263

3000

30000

50.1

>>600

4500

During this time the Apache server is taking up an entire CPU core. This doesn’t stop even after start job times out.

Has anyone seen this before? Any thoughts as to what I could try? I’ve attached a protocol showing the steps I can to connect to the SOAP service. You won’t be able to run it, but it gives an overview of what I’m trying to do.

Thanks

Joe

Historic PostHistoric Post (BIOVIA) 

Hi Joe,

I haven't had a chance to run this yet, but what version of AEP / PP are you using? Also, you're saying that the httpd.exe process (not scisvr.exe) is taking-up the CPU, right?

-Jason

Joe_BJoe_B

Thanks Jason,

I'm running aep9.1 HF2 on windows server 2008. My attached protocol won't run for you, but it gives you an idea of the steps and how I've used the soap connectors. You should see the effect if you create a simple soap service that does nothing and use the login and then start job to upload a file. increasing file sizes create a disproportionally large delay. If you don't see the effect I'll mock up a simpler couple of protocols showing the effect that can easily be added and executed on your server.

Regards

Joe.

Joe_BJoe_B

Hi Jason,

I've attached 2 more example protocols that you should be able to run. These are much simpler and still show the issue.

-SOAP wrapper issue demo.xml is the target protocol.

-Example for forum2.xml is the protocol that calls the wrapper protocol.

Save the wrapper to the webserices area somewhere, inspect the webservice to get the polling WSDL and then edit the SOAP connectors in the Example for forum2 to point to the WSDL on your server.

Enter your username and password. Then run it with differing number fo records. Again time taken to upload and download a file is exponential compared to the size instead of linear. Some example timings are below (in seconds)

recordsstartreturn
1000.060.09
10000.091.05
100005.7970
2000022.7>300

Thanks

Joe

Joe_BJoe_B

Hi Jason,

Thanks for your reply on job pooling - thats been very helpful already.

Regarding this SOAP issue. We've done some more testing, calling the same SOAP service from the JAVA SDK. The SDK does not see the exponential like time increase that we are seeing when using the SOAP connectors in Pipeline Pilot. This further makes me beleive there is some kind of bug in the AEP SOAP connectors (or as likely that I'm doing something wrong).

See timings below when using the Java SDK

Recordsstartreturn
1000.2s0.4s
10000.5s2.1s
100000.5s8.2s
200001.2s16.7s

When you compare these to the ones above, you see the clear difference.

Have you had a chance to look at what's happening?

Thanks

Joe

Joe_BJoe_B

To close out this thread. There is a problem and its been raised as bug No PPC-6849

Kip ShafferKip Shaffer
I too have found that there is a huge penalty for using SOAP when the size of the returned value is significant. Furthermore, the degree of complexity of the return value also has a big effect.

I used SoapUI as an independent client to call the Pipeline Pilot 9.2 protocol.  The protocol returned 2.5MB block of XML results.  The protocol runs in 1 second, but took 30s to return the SOAP response.

I changed the return value to JSON.  That reduced the time for the SOAP response down to just 7.5s, a marked improvement.

The final test demonstrates that the overhead is tied to using SOAP as the calling mechanism. A REST interface was defined to call the test protocol. This time, the response was received in approximately 1s. This completely eliminated any time penalty associated with the remote call.

I conclude from these experiments that it can take a significant time to encode the response to be safe for embedding in the SOAP's XML response. Using the REST protocol bypasses any problems by eliminating this encoding step. I recommend using REST for any web service that returns a significant amount of data. Though, in Pipeline Pilot, defining a new REST endpoint requires some administrative effort.
KristerKrister (Symyx Technologies, Inc.) 
Thanks for your investigation and the suggestion to use REST in favour of SOAP.

I would also like to add that that the above mentioned issue, PPC-6849, was fixed in the 2016 release of BIOVIA Pipeline Pilot Server:
PPC-6849
Fixed a significant performance problem with sending text data containing XML characters as parameter values to a protocol launched via polling web services mode (WSDL)