java - File descriptor leak when getting response code. (cxf, ssl) -

we have problem in production quite while now...

this follow of: my other question way more specifics think posting new question justified (if not i'll add info other question).

here goes:

so, have file descriptor leak on aix (ibm) java 6 weblogic using application using cxf , address web service of our own jsb routes towards our ws.

when using file leak detector agent in startup of weblogic , dumping getcurrentopenfiles() , filtering on listener.socketrecord programatically have 2000+ open sockets;

these java sockets , file descriptors, net-sockets (viewed netstat) closed, time, programmatic ones (and ones viewed

lsof -p $pid_of_managed_server 2> /dev/null|grep tcp|wc -l 

)stay open (and cause many open files issue)).

this head of stack of 1 of open file descriptors inside jvm:

record socket tst-cjcsr.just.fgov.be/10.239.7.19:443 thread:[active] executethread: '0' queue: 'weblogic.kernel.default (self-tuning)' on thu nov 10 10:25:56 cet 2016     @ java.net.plainsocketimpl.create(plainsocketimpl.java:188)     @ java.net.socket.createimpl(socket.java:411)     @ java.net.socket.connect(socket.java:544)     @ weblogic.net.http.httpsclient.openwrappedsslsocket(httpsclient.java:565)     @ weblogic.net.http.httpsclient.openserver(httpsclient.java:296)     @ weblogic.net.http.httpsclient.openserver(httpsclient.java:373)     @ weblogic.net.http.httpsclient.new(httpsclient.java:528)     @ weblogic.net.http.httpsurlconnection.connect(httpsurlconnection.java:239)     @ weblogic.net.http.httpurlconnection.getinputstream(httpurlconnection.java:409)     @ weblogic.net.http.soaphttpsurlconnection.getinputstream(soaphttpsurlconnection.java:37)     @ weblogic.net.http.httpurlconnection.getresponsecode(httpurlconnection.java:1038)     @ org.apache.cxf.transport.http.urlconnectionhttpconduit$urlconnectionwrappedoutputstream.getresponsecode(urlconnectionhttpconduit.java:266)     @ org.apache.cxf.transport.http.httpconduit$wrappedoutputstream.doprocessresponsecode(httpconduit.java:1550)     @ org.apache.cxf.transport.http.httpconduit$wrappedoutputstream.handleresponseinternal(httpconduit.java:1579)     @ org.apache.cxf.transport.http.httpconduit$wrappedoutputstream.handleresponse(httpconduit.java:1520)     @ org.apache.cxf.transport.http.httpconduit$wrappedoutputstream.close(httpconduit.java:1317)     @ org.apache.cxf.transport.abstractconduit.close(abstractconduit.java:56)     @ org.apache.cxf.transport.http.httpconduit.close(httpconduit.java:632)     @ org.apache.cxf.interceptor.messagesenderinterceptor$messagesenderendinginterceptor.handlemessage(messagesenderinterceptor.java:62)     @ org.apache.cxf.phase.phaseinterceptorchain.dointercept(phaseinterceptorchain.java:272)     @ org.apache.cxf.endpoint.clientimpl.doinvoke(clientimpl.java:572)     @ org.apache.cxf.endpoint.clientimpl.invoke(clientimpl.java:481)     @ org.apache.cxf.endpoint.clientimpl.invoke(clientimpl.java:382)     @ org.apache.cxf.endpoint.clientimpl.invoke(clientimpl.java:335)     @ org.apache.cxf.frontend.clientproxy.invokesync(clientproxy.java:96)     @ org.apache.cxf.jaxws.jaxwsclientproxy.invoke(jaxwsclientproxy.java:136)     @ com.sun.proxy.$proxy380.requestcriminalrecord(unknown source)     @ be.fgov.just.cjr.application.dossier.dossierbean$1.call(dossierbean.java:225) 

i can imagine applicative issue because can't find other cases online similar defects.

does can make more sense stack?

for instance: strikes me odd httpconduit.close() wants create connection...

another point: issue doesn't occur same stack of technologies non-https calls. (which makes sense because stacktrace mentions httpsclient)

cxf version: 2.7.18

cxf config:

<beans xmlns="http://www.springframework.org/schema/beans"        xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"        xmlns:jaxws="http://cxf.apache.org/jaxws"        xmlns:soap="http://cxf.apache.org/bindings/soap"        xmlns:cxf="http://cxf.apache.org/core"        xmlns:context="http://www.springframework.org/schema/context"        xsi:schemalocation="             http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd             http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd             http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd             http://cxf.apache.org/jaxws http://cxf.apache.org/schemas/jaxws.xsd">     <import resource="classpath:meta-inf/cxf/cxf.xml"/>      <context:property-placeholder location="file:${config.file.location}cjr-extract-${cjr.environment}.properties"/>      <jaxws:client id="cjcscgservice"                   serviceclass="be.fgov.just.cjr.ws.extract.generated.cjcscgservice"                   address="${ws.extract.wsdl.endpoint}">         <jaxws:binding>             <soap:soapbinding version="1.2"/>         </jaxws:binding>     </jaxws:client>      <cxf:bus>         <cxf:outinterceptors>             <bean class="our.package.interceptors.someinterceptor"                   id="websecurityinterceptor">                 <constructor-arg>                     <map>                         <entry key="action" value="timestamp signature"/>                         <entry key="user" value="${org.apache.ws.security.crypto.merlin.keystore.alias}"/>                         <entry key="passwordcallbackref">                             <ref bean="passwordcallback"/>                         </entry>                         <!--entry key="signaturepropfile" value="properties/our.properties"/-->                         <entry key="signaturepropfile" value="file:${location}/our.properties"/>                         <entry key="signaturekeyidentifier" value="directreference" />                     </map>                 </constructor-arg>             </bean>         </cxf:outinterceptors>         <cxf:properties>             <entry key="signatureparts"                    value="{element}{http://www.w3.org/2003/05/soap-envelope}body;{element}{http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd}timestamp"/>         </cxf:properties>     </cxf:bus>      <bean id="passwordcallback" class="our.package.authentication.passwordcallbackhandler">         <property name="password" value="${password}"/>         <property name="alias" value="${org.apache.ws.security.crypto.merlin.keystore.alias}"/>     </bean> </beans> 

java:

java version "1.6.0" java(tm) se runtime environment (build pap3260sr15fp1-20140110_01(sr15 fp1)) ibm j9 vm (build 2.4, jre 1.6.0 ibm j9 2.4 aix ppc-32 jvmap3260sr15- 20131231_180656 (jit enabled, aot enabled) j9vm - 20131231_180656 jit  - r9_20130920_46510ifx3 gc   - ga24_java6_sr15_20131231_1152_b180656) jcl  - 20140107_01 

weblogic 10.3.6

thanks hints, pointers, and, of course - if possible -, answers ;-)

s.

ejp requested code, here is:

the call itself:

private cjcscgservice cjcscgservice;  private callable<requestcriminalrecordresponse> callservice(final requestcriminalrecordrequest request) {     return new callable<requestcriminalrecordresponse>() {         @override         public requestcriminalrecordresponse call() throws exception {             try {                 return cjcscgservice.requestcriminalrecord(request); // line 225.             } catch (exception e) {                 facesmessages.error("technicalerror");                 log.error("encountered technical error", e);                 return null;             }         }     }; } 

the generated webservice:

package be.fgov.just.cjr.ws.extract.generated;  import javax.jws.webmethod; import javax.jws.webparam; import javax.jws.webresult; import javax.jws.webservice; import javax.jws.soap.soapbinding; import javax.xml.bind.annotation.xmlseealso;   /**  * class generated jax-ws ri.  * jax-ws ri 2.1.7-b01-  * generated source version: 2.1  *   */ @webservice(name = "cjcscgservice", targetnamespace = "http://secret/service-v1.0-rc2") @soapbinding(parameterstyle = soapbinding.parameterstyle.bare) @xmlseealso({     objectfactory.class }) public interface cjcscgservice {     /**      *       * @param requestcriminalrecordrequest      * @return      *     returns be.fgov.just.cjr.ws.extract.generated.requestcriminalrecordresponse      */     @webmethod(action = "http://secret/service/requestcriminalrecord")     @webresult(name = "requestcriminalrecordresponse", targetnamespace = "http://secret/service-v1.0-rc2", partname = "requestcriminalrecordresponse")     public requestcriminalrecordresponse requestcriminalrecord(         @webparam(name = "requestcriminalrecordrequest", targetnamespace = "http://secret/service-v1.0-rc2", partname = "requestcriminalrecordrequest")         requestcriminalrecordrequest requestcriminalrecordrequest); } 

with:

<http-conf:conduit name="*.http-conduit">     <http-conf:client connection="close" /> </http-conf:conduit> 

the connections (os level) close; still have leak (behavior stated above).

with:

<http-conf:conduit name="*.http-conduit">     <http-conf:client connection="keep-alive" /> </http-conf:conduit> 

both connections (os level) file descriptors keep going up...

(additional info: os level connections go down (after time out guess) file descriptors stay open...)

from perspective problem related cxf configuration, perhaps interplay of weblogic + cxf issue, because it's cxf component manages connection (see stacktrace). suggested trying different keep-alive settings (how set keep-alive described here). seems didn't beat open file descriptors though.

update:

after thinking more problem:

at org.apache.cxf.transport.http.urlconnectionhttpconduit$urlconnectionwrappedoutputstream.getresponsecode(urlconnectionhttpconduit.java:266) @ org.apache.cxf.transport.http.httpconduit$wrappedoutputstream.doprocessresponsecode(httpconduit.java:1550) @ org.apache.cxf.transport.http.httpconduit$wrappedoutputstream.handleresponseinternal(httpconduit.java:1579) @ org.apache.cxf.transport.http.httpconduit$wrappedoutputstream.handleresponse(httpconduit.java:1520) @ org.apache.cxf.transport.http.httpconduit$wrappedoutputstream.close(httpconduit.java:1317) 

seems when closing http connection cxf tries response code , creates socket that, , sockets wait response never comes indefinitely creates leak. whole flow described here under simplified client workflow section. so, next question is, tst-cjcsr.just.fgov.be ever reply? and, next suggestion configure receivetimeout http-conf:client described in link above.