CFFILE vs Java.IO.BufferedWriter

Jul 21, 2006

I do not often use the CFFILE tag to write out large amounts of data, typically I only use it for adding a line to a log file or some other relatively small task. Recently however, I needed to create a process to write out large text files (approximately 5.5 MB, 17,500 lines of text) and I quickly discovered that the CFFILE tag is extremely inefficient for such a purpose. I had a simple loop that concatenated the text together and then used the CFFILE tag to write the file at the end. On average it was taking about 600 seconds to create the file. Once I started to think about it, the problem became very apparent. Every call using the CFFILE tag to write to a file does the following:

• open the file
• read the file
• add the new text
• save the file
• close the file

All this is running through ColdFusion to the underlying Java classes so there is a lot of overhead and excess processing in all the running back and forth. Time to find a better solution.

Using CFFILE to write the file line by line.

My first idea was to break up the amount of work the CFFILE command had to do into smaller chunks, write the file one line at a time.

<cfset CrLf = Chr(13)>

<cffile action="write" file="#logFilePath#" output="##Software: Microsoft Internet Information Services 6#CrLf#" nameconflict="overwrite">
<cffile action="append" addnewline="yes" file="#logFilePath#" output="##Version: 1.0#CrLf#">
<cffile action="append" addnewline="yes" file="#logFilePath#" output="##Date: #selectLogFile.log_stamp[1]##CrLf#" >
<cffile action="append" addnewline="yes" file="#logFilePath#" output="##Fields: date time cs-method cs-uri-stem c-ip cs(User-Agent) cs(Cookie) cs(Referer) cs-host#CrLf#">

<cfset logData = '>

<cfloop from="1" to="#selectLogFile.RecordCount#" index="i">
	<cffile action="append" file="#logFilePath#" output="#selectLogFile.log_entry[i]##CrLf#">
</cfloop>

Total time to create the file (in milliseconds)

• Pass 1 - 138,852
• Pass 2 - 137,586
• Pass 3 - 139,946

Average time to create the file was almost 140 seconds. Better than 400 seconds but still insane!

Using CFFILE and 'buffering' the output

Building on my first idea, I decided to try and buffer the output data, this would reduce the number of times the CFFILE tag would need to be called. I set a variable called logData and filled it with twenty lines of text before writing out the data

<cfset CrLf = Chr(13)&Chr(10)>

<cffile action="write" file="#logFilePath#" output="##Software: Microsoft Internet Information Services 6#CrLf#" nameconflict="overwrite">
<cffile action="append" addnewline="yes" file="#logFilePath#" output="##Version: 1.0#CrLf#">
<cffile action="append" addnewline="yes" file="#logFilePath#" output="##Date: #selectLogFile.log_stamp[1]##CrLf#" >
<cffile action="append" addnewline="yes" file="#logFilePath#" output="##Fields: date time cs-method cs-uri-stem c-ip cs(User-Agent) cs(Cookie) cs(Referer) cs-host#CrLf#">

<cfset logData = '>

<cfloop from="1" to="#selectLogFile.RecordCount#" index="i">
		<cfif i MOD 20 EQ 0> 
				<cffile action="append" file="#logFilePath#" output="#logData#">
				<cfset logData = '> 
		</cfif>					
		<cfset logData = logData & selectLogFile.log_entry[i] & CrLf>
</cfloop>

<cfif logData NEQ '> 
	  <cffile action="append" file="#logFilePath#" output="#logData#">
</cfif> 

Total time to create the file (in milliseconds)

• Pass 1 - 10,438
• Pass 2 - 12,063
• Pass 3 - 10,876

Average time to create the file was down to around 11 seconds. Much better but 11 seconds still seemed like a long time to write out the file. I tried different buffer sizes but the processing time was still much longer than I wanted.

Writing with Java.IO

I came to the conclusion that the only way to really speed up the process was to directly access the Java.IO classes from Coldfusion. It was time to put on the Java hat. Java has a slew of input/output file operation classes, I knew my solution was there ... somewhere.

<cfset logFile = CreateObject('java', 'java.io.File').init(logFilePath)>
<cfset fw = CreateObject('java', 'java.io.FileWriter').init(logFile)>
<cfset bw = CreateObject('java', 'java.io.BufferedWriter').init(fw, 4096)>

<cfset CrLf = Chr(13)&Chr(10)>

<cfset res = bw.write('##Software: Microsoft Internet Information Services 6' & CrLf)>
<cfset res = bw.write('##Version: 1.0' & CrLf)>
<cfset res = bw.write('##Date: #selectLogFile.log_stamp[1]#' & CrLf)>
<cfset res = bw.write('##Fields: date time cs-method cs-uri-stem c-ip cs(User-Agent) cs(Cookie) cs(Referer) cs-host' & CrLf)>

<cfloop from="1" to="#selectLogFile.RecordCount#" index="i">
	<cfset res = bw.write(selectLogfile.log_entry[i] & CrLf)>

</cfloop>
<!--- output any remaining data in the buffer ---->
<cfset res = bw.flush()>
<cfset res = bw.close()>

Total time to create the file (in milliseconds)

• Pass 1 - 1,563
• Pass 2 - 1,062
• Pass 3 - 1,094

Average time to create the file was down to an average of about 1 second.

Conclusion

600 seconds vs 1 second, hmmmm, I think I found a winner. The CFFILE is a very handy tag and saves having to write the extra lines of code for opening/closing etc. for simple file operations, however there is a huge performance hit when working with large files and having to use the CFFILE tag in an iterative process. As with any problem, there are always multiple solutions, pros and cons to each. For writing large files, the pros of the Java method significantly outweighed any perceived cons.