Workaround for CFHTTP and Compressed HTTP Response from IIS

Viewing By Entry

Workaround for CFHTTP and Compressed HTTP Response from IIS

Some versions of IIS may use an HTTPCompression algorithm that is incompatible with ColdFusion (or Java in general). When using CFHTTP to performthe operations GET or POSTon the target website, if the target site is using Microsoft IIS with HTTP Compression enabled, the CFHTTP tag may report a Connection Failure. If the packets are traced with Ethereal, you might even see that all the compressed data is returned to the CFHTTP tag in the response, and the status code might even be 200 OK. If you are getting a Connection Failure result, the problem is that ColdFusion doesn't know how to deflate the compressed content. If this situation occurs, then you can specify additional headers to be sent with the CFHTTP request which tell the target webserver to disable compression for that response.

Thanks to Dan Switzer for adding a comment which suggests using an Accept-Encoding value of * instead of deflate;q-0. The * worked for me in situations where deflate did not.

The above headers, when used in cfhttpparam, will request that the target webserver send an uncompressed response. The headers use a q parameter to specify the quality factor as described in RFC 2616.

I couldn't reproduce this using CFMX 6.1 on Windows XP connecting to IIS 6 on Win2k3 with HTTP Compression turned on for application and static files. Not all versions ofIIS mayhave this problem, and perhaps it was fixed in recent IIS versions. In my case, I confirmed that IIS was sending compressed content by verifying that the compressed output was stored in the IIS Temporary Compressed Files directory and a packet sniff showed the compressed content as gibberish. CFHTTP was able to write out the cfhttp filecontent successfully, and reviewing that output confirmed that the content was properly deflated and readable. Here is that HTTP Request/Response from CFHTTP without using the headers described above.

GET /serat/cgi.htm HTTP/1.1
Host: ps-win2k3svr1
Connection: close, TE
TE: trailers, deflate, gzip, compress
User-Agent: ColdFusion
Accept-Encoding: deflate, gzip, x-gzip, compress, x-compress

HTTP/1.1 200 OK
Content-Length: 192443
Content-Type: text/html
Content-Encoding: gzip
Content-Location: http://ps-win2k3svr1/serat/cgi.cfm
Last-Modified: Wed, 28 Jul 2004 14:16:51 GMT
Accept-Ranges: bytes
ETag: "80cb9f86ad74c41:4aa"
Vary: Accept-Encoding
Server: Microsoft-IIS/6.0
Date: Wed, 28 Jul 2004 14:34:26 GMT

{lots of encoded data removed here}

posted on 28 July, 2004 at 11:43 AM.
ColdFusion | Comments (25)

Comments

Steve.

thank you for this information, you just made my night!!! i was getting the connection failure, from a site i could browse all day long from my browser. anyway, good stuff.

thanks!!!

comment by Tony Weeg, posted at 8/19/04 6:38 PM

Steven: A little additional info about this problem:

http://admin.mxblogspace.journurl.com/?mode=article&entry=3178

comment by Roger Benningfield, posted at 8/26/04 1:55 PM

You rock--I was pulling my hair out trying to find out why my code that used to work no longer worked. I guess I still don't know (although I suspect the answer is that the server I'm hitting now compresses the response by default whereas it did not when the code was working originally) but no matter--using your s got me back up and running. Thanks!

Josh

comment by Josh Adams, posted at 1/17/05 9:34 PM

I can't thank you enough! This just saved my hide on a huge project. Any idea if this has been posted as a bug for Macromedia to look at? In my circumstance, I encountered this error with both CFMX6.1 and CFMX7.

-Nelson

comment by Nelson Winters, posted at 7/5/05 8:02 PM

Thanks for the info. Not sure if it will answer my problem, but I am getting the same error message when trying to dump a url content from a server. I do not manage the target site.

The url is an https, so I may be getting a security error instead.
Here is the code that I am using (sans url for security purposes)

Does this look right? Thanks!

comment by John Wells, posted at 7/13/05 7:05 AM

my code seems to have been stripped out.
here it is with substitute chars
[CFHTTP METHOD="Get" URL="https://MyURLpath" resolveurl="yes" ]
[cfhttpparam type="Header" name="Accept-Encoding" value="deflate;q=0"]
[cfhttpparam type="Header" name="TE" value="deflate;q=0"]
[/cfhttp]
thanks

comment by John Wells, posted at 7/13/05 7:15 AM

Once again, thanks for a great article Steven! I have been trying to communicate with a SOAP webservice from Tourico and ran into the same HTTPCompression algorithm problem. After adding the cfhttpparams you suggested, everything worked perfect! FYI I am running CFMX 7 on Win2k3.

comment by David Day, posted at 8/17/05 8:56 AM

Hi,

I'm having some problems using cfhttp and tried out your fix here but it didnt seem to work. On our development machine we have both CF 5.5 and CFMX 7 setup as virtual machines so we can test our code in both (we are moving to MX7). I'm trying to get something working on CFMX 7 which uses cfhttp. The code I have works fine in CF 5.5 but I just get a connection failure in CFMX. In fact, I cant even run [cfhttp method="GET" url="http://www.ubc.ca/about/index.html"] on CFMX. Even that gives a connection failure. Any ideas here? Its taking about 1.5 or 2 seconds to give the error so I don't think its timing out...

comment by dawn, posted at 8/18/05 1:49 PM

Very helpful post - thanks so much for putting this info online. There's precious little other info about this out there.

I've clicked on a few of your sponsor links by way of a small thanks!!

Nick

comment by Nick Thompson, posted at 10/19/05 7:42 AM

THANK YOU SO MUCH FOR THIS!!

I had to say that in caps cuz this solution is perfect - thank you very much!

comment by Derek T. Versteegen, posted at 12/1/05 12:39 PM

Thanks bunches this was the culprit for me. I was using the gzip.ddl in iis6 on w2k3 std with mx 7.0.1. I could see the source info was gibberish. cfhttp problem with flashpaper, and cfdocument solution.

Thanks again!!!!

comment by John, posted at 1/4/06 12:26 AM

Bravo! That totally works.

comment by theEye, posted at 2/16/06 10:03 PM

I've been playing around with Port80 Software's httpZip product on my development server and was having a similiar experience with CFHTTP. My problem differs in that the very first request to a URL on my server (via a CFHTTP request) works fine, but subsequent hits return the "Connection Failure" error.

Turns out the problem can be fixed in a number of ways:

1) When configuring httpZip, set up all compressed mime types to ignore anything with the header "ColdFusion". This will fix the issue with any default CFHTTP calls (Schedule Tasks, CFCACHE, etc.) If you're manually setting the User Agent string, see one of the following solutions.

2) Pass in a <cfhttpparam type="header" name="Accept-Encoding" value="*" />. For some reason Steve's suggestion does not work, but passing in the asterisks value does work. I was never able to get a "Connection Failure" message when adding this header information.

3) The last method is to add custom header to the page serving up content, which tells the httpZip ISAPI filter to ignore compressing the request: <cfheader name="httpZip" value="no-compression" />. For example, you could have all your CFML pages look for the custom header "Gzip-Disabled" and if the header exists and is true, you could then execute the CFHEADER tag to tell the ISAPI filter not to compress the page.

-Dan

comment by Dan G. Switzer, II, posted at 3/13/06 4:59 PM

Thanks for this info! Very helpful.

comment by David, posted at 4/7/06 7:44 PM

Good info. I am running into a similar issue with web services, where my cfinvoke is going to a .aspx page that has http compression on it. I have tried everything I can come up with including manually adding the headers you use for cfhttp but with addSOAPRequestHeader (in CFMX7.02). Have any magic words for making cfinvoke work with http compression enabled on the server ?

comment by Andrew, posted at 10/20/06 12:58 AM

One word: Awesome....

My XML API with our ecommerce vendor, after working for 2+ years broke 3 months ago, and we haven't been able to figure it out. I am running CFMX 7 on IIS 6, and connecting to an IIS 6 server for the API. Adding the two CFHTTPPARAMs for the compression fixed the problem.

THANK YOU THANK YOU THANK YOU

comment by Avi, posted at 12/5/06 8:15 PM

Another brilliant Coldfusion blog post that has sorted out my problem in minutes. Thanks for much for that Steven.

This problem is obviously something that isn't about to go away. Using CFMX 7.1 and got the 'connection failed' back when using CFHTTP to connect to a Microsoft Commerce Server website. Baffled me until I tracked down your post.

Thanks again.

comment by James Allen, posted at 4/22/07 7:45 AM

Does anyone know how to work around this problem in a scheduled task? Since it uses CFHTTP under the hood, the same problem applies. But I don't know of any way to coax the scheduled task into accepting the compressed content. Any ideas?

comment by Brian Kotek, posted at 4/27/07 1:16 AM

Brian,
Instead of targeting the URL on a server which uses header compression, couldn't you just call a CFM template on a different server which then executes the CFHTTP command with the relevant headers to get the compressed data to download? So your not actually using the Coldfusion schedulers CFHTTP but your own custom version.

comment by James Allen, posted at 4/27/07 4:15 AM

Unfortunately I don't have a separate CF server that I can disable compression on just to handle the redirected schedule call.

comment by Brian Kotek, posted at 4/27/07 11:15 AM

Possible Red Herring:
I have MX7 on Win 2003 and was getting "Connection Failure: Status code unavailable" and thought that since the file being picked up was xml and was gziped according to the Content encoding (by live headers plugin on firefox) that this would have been fixed with the fix mentioned here (as we have used this fix with success for something else in the past).
However this was not the case it turned out that all that was the problem was the directory given in the path attiribute of cfhttp didnt exist ...doh! A ridiculous error message for the fault really.

So watch out if the above fix didnt work for you.

comment by Adrian, posted at 5/10/07 12:29 PM

Brian,

A compromise solution that may work for your schedule task problem is to turn off the DEFLATE compress in IIS but leave on GZIP. Something like this will set the DEFLATE to only .dll files (the default) assuming you aren't scheduling .dll files:

cscript.exe C:\Inetpub\AdminScripts\adsutil.vbs set W3Svc/Filters/Compression/DEFLATE/HcScriptFileExtensions "dll"

comment by Colby, posted at 5/15/07 11:04 AM

ColdFusion 9 has added a new "compression" attribute, but it is not documented in the Help files. You will see it in the autocomplete of CFBuilder. You should use the attribute in CF9 rather than the workaround suggested here for earlier versions.

comment by Steven Erat, posted at 1/14/10 9:27 AM

I had a similar issue with CFHTTP/HTTP Response from IIS when I applied CHF 2 or 3 to ColdFusion 8.0.1. Turns out it's a bug they introduced while trying to fix the very same problem. Here's more on the issue:

http://www.brooks-bilson.com/blogs/rob/index.cfm/2...

comment by Rob Brooks-Bilson, posted at 1/14/10 9:31 PM

thanks man, you're the man!

tq tq tq

comment by nickmenza, posted at 2/9/10 3:51 AM

Calendar

Search This Site

This is an exact search only

About This Site

I live west of Boston and work as a Software Engineer with ColdFusion and Flex, specializing in Linux. Recently I graduated in Professional Digital Photography from CDIA.