Title: wrong handling of append only files
Type: behaviour Severity: normal
Components: Core Versions: 2.5.1
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pjenvey Nosy List: fwierzbicki, gaddo, pjenvey, zyasoft
Priority: urgent Keywords:

Created on 2009-09-13.14:38:29 by gaddo, last changed 2010-03-21.17:32:01 by pjenvey.

msg5144 (view) Author: (gaddo) Date: 2009-09-13.14:38:28
jython does not respect the append flag to 
open, causing data loss.

this is on linux.

stracing jython, i can se that the file
is opened O_APPEND in cpython but is not
with jython 

i discovered this bug while trying to run
mercuria with jython.

gaddo@enterprise:~/jython$ ./jython 
Jython 2.5.0 (Release_2_5_0:6476, Jun 16 2009, 13:33:26) 
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.6.0_0
Type "help", "copyright", "credits" or "license" for more information.
>>> fd = open('test', 'ab')
>>> fd.write('test1\n')
>>> fd2 = open('test','ab')
>>> fd2.write('test2\n')
>>> quit()
gaddo@enterprise:~/jython$ cat test
msg5145 (view) Author: Jim Baker (zyasoft) Date: 2009-09-13.22:43:13
May require RC3
msg5146 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2009-09-13.22:50:15
Agreed, nothing is worse than potential data loss -- and my first
response -- just use close() -- looks insufficient in this case --
pjenvey: what do you think?
msg5147 (view) Author: Jim Baker (zyasoft) Date: 2009-09-13.23:02:59
Repeated bug on OS X 10.5.8 on trunk, including using close.
msg5148 (view) Author: Philip Jenvey (pjenvey) Date: 2009-09-13.23:16:10
For this smaller example you just need to fd.flush() or close() in line 
3 to get the right output.

The bigger problem is that 'a' mode doesn't use O_APPEND, which makes 
writes always append to EOF regardless of the fd's position. You could 
use it to open a log file twice, for example, and both fds would always 
append to the end (though in reality I'm not sure how safe that actually 
is when files are buffered by the underlying OS)

So the JVM provides one single way of getting a file in O_APPEND, via 
FileOutputStream(file, appending=True). Unfortunately that fd is opened 
write only, so we couldn't support the 'a+' mode with it
msg5149 (view) Author: Philip Jenvey (pjenvey) Date: 2009-09-13.23:26:40
Unfortunately I guess all we can do is fake 'a+' by seek()ing to the end 
before every write.. which is pretty lame
msg5150 (view) Author: Philip Jenvey (pjenvey) Date: 2009-09-14.01:08:22
applied said workarounds in r6796. It's a bit invasive so we need an rc3
msg5151 (view) Author: (gaddo) Date: 2009-09-14.06:11:26
i am worried that this can fail 
in a multithreaded program, where 2 threads
append to the same log.
if both threads seek and then write, they can 
see the same file length, and overwrite
each other data.

maybe this is not a problem with jython
right now ...
msg5152 (view) Author: Philip Jenvey (pjenvey) Date: 2009-09-14.06:30:08
with the fix I applied it's still a potential issue in 'a+' mode. 
Unfortunately there's nothing else we can really do about that except 
document the problem

However regular 'a' mode is now safe as it's using O_APPEND

Also it's not a problem at all if threads are writing to the same file 
object, which I would recommend. Python file operations are atomic
msg5581 (view) Author: Philip Jenvey (pjenvey) Date: 2010-03-21.17:32:01
This fix introduced a regression in 2.5.1, see #1576
Date User Action Args
2010-03-21 17:32:01pjenveysetmessages: + msg5581
2009-09-14 06:30:08pjenveysetmessages: + msg5152
2009-09-14 06:11:27gaddosetmessages: + msg5151
2009-09-14 01:08:22pjenveysetstatus: open -> closed
resolution: fixed
messages: + msg5150
2009-09-13 23:26:40pjenveysetmessages: + msg5149
2009-09-13 23:16:11pjenveysetmessages: + msg5148
2009-09-13 23:02:59zyasoftsetmessages: + msg5147
2009-09-13 22:50:15fwierzbickisetmessages: + msg5146
2009-09-13 22:48:49pjenveysetassignee: pjenvey
2009-09-13 22:43:14zyasoftsetpriority: urgent
nosy: + fwierzbicki, pjenvey, zyasoft
messages: + msg5145
versions: + 2.5.1, - 2.5b1
2009-09-13 14:38:29gaddocreate