OVITO scripting errors out on image 1001 with "too many open files"

[steve]

Hi - Trying to run OVITOs 2.9 (stable) in scripting mode
to render one image per dump snapshot.  The snapshots are
in different LAMMPS dump files, and each is a zoom in on a different region
of a big simulation (over time), so each snapshot is effectively unrelated to
the others.  The rendering portion of my Python
script is below.  Along with the error that occurs after
1000 images have been output.

1000 is the default limit for my Linux box as to how
many files can be open at the same time.  So apparently
OVITO is keeping each file open.  Is there a way to tell
it to close the snapshot file (each loop iteration), or maybe a better way to
handle this workflow with OVITO?

Thanks,
Steve

------------------------

# main loop over dump files

for i,infile in enumerate(files):
  node = ovito.io.import_file(infile)
  node.add_to_scene()
  node.source.particle_properties.position.display.radius = radius
  if GetImage == 1:
    outfile = "%s/%s.png" % (outdir,os.path.basename(infile))
    print("%s --> %s" % (infile,outfile))
    vset = RenderSettings(filename=outfile,size=(512,512))
    #vset.renderer = TachyonRenderer()    # hi-res
    #vset.renderer.shadows = True
    vset.renderer = OpenGLRenderer()    # lo-res
    vp.zoom_all()
    node.compute()
    vp.render(vset)
  node.remove_from_scene()


ERROR: Failed to open input file: Too many open files
Traceback (most recent call last):
  File "process_render.py", line 101, in <module>
  File "/home/sjplimp/tools/ovito-2.9.0-x86_64/bin/../lib/ovito/plugins/python/ovito/io/__init__.py", line 119, in import_file
  File "/home/sjplimp/tools/ovito-2.9.0-x86_64/bin/../lib/ovito/plugins/python/ovito/__init__.py", line 173, in _ObjectNode_wait
RuntimeError: Data pipeline evaluation failed with the following error: Failed to open input file: Too many open files

[Alexander Stukowski]

Steve,

This may or may not be a bug in Ovito. Normally, it shouldn't keep any file handles open. I will look into this myself.

You can try to rearrange the loop as follows:

Code: [Select]

node = ovito.io.import_file(files[0])
node.add_to_scene()
node.source.particle_properties.position.display.radius = radius
vset = RenderSettings(size=(512,512))
#vset.renderer = TachyonRenderer()    # hi-res
#vset.renderer.shadows = True
vset.renderer = OpenGLRenderer()    # lo-res

for i,infile in enumerate(files):
  node.source.load(infile)
  if GetImage == 1:
    outfile = "%s/%s.png" % (outdir,os.path.basename(infile))
    vset.filename=outfile
    print("%s --> %s" % (infile,outfile))
    vp.zoom_all()
    #node.compute()  <-- not needed
    vp.render(vset)

Not sure if it will help, but in principle the call to FileSource.load() should ensure that the file reader is being re-used.

[Alexander Stukowski]

I was able to reproduce the problem. Unfortunately, it turned out that the rearrangement of the for-loop doesn't solve the problem.

This issue seems to be caused by a bug in the GSD library, which is used by OVITO to read GSD simulation files (produced by the HOOMD code). Even though you are loading files with a different format, OVITO still tries to open them with the GSD I/O functions as part of the format auto-detection routine. Apparently, the GSD I/O function doesn't close the file handle properly in case it is not a GSD file.

I will try to update the copy of the GSD library shipping with Ovito to resolve this issue. In the meantime, you can only work around the problem by hacking the FileSource.load() Python function of Ovito. For that, open the file ovito-2.9.0-x86_64/lib/ovito/plugins/python/ovito/io/__init__.py in a text editor and go to line 143. Replace the following three lines

Code: [Select]

    importer = FileImporter.autodetect_format(self.dataset, location)
    if not importer:
        raise RuntimeError("Could not detect the file format. The format might not be supported.")

with the this code

Code: [Select]

    importer = self.importer

This disables the format auto-detection and Ovito will assume that subsequent files always have the same format as the file loaded initially. Note that you need to work with the rearranged for-loop I posted earlier.

[steve]

That __init__.py edit fixed it - script is chugging
past 1000 files now ...

Thanks!
Steve

[Alexander Stukowski]

Ok, great.

I have reported the problem to the upstream developers of the GSD I/O layer: https://bitbucket.org/glotzer/gsd/issues/13/file-handle-not-closed-after-failed-call

In the meantime, I also implemented a workaround for the bug in Ovito. Future versions of the program will no longer suffer from this issue.

-Alex

[Alexander Stukowski]

The file descriptor bug has been resolved by the upstream GSD developer and I have built a new development release of Ovito 3 (3.0.0-dev180), which is available on the download page.

In case you want to switch to the 3.x release already: The existing Python script should still work, but the backward compatibility layer may not be perfect. It is advisable to adapt the script to the new programming interface of Ovito 3.x. Here is a template:

Code: [Select]

from ovito.io import import_file
from ovito.vis import *
from ovito.data import *
import sys
import os

files = sys.argv[1:]
outdir = "output"

# Import data of first frame
pipeline = import_file(files[0])
pipeline.add_to_scene()

# Configure visual appearance:
pipeline.source.particles['Position'].vis.radius = 1.0
pipeline.source.expect(SimulationCell).vis.line_width = 0.3

# Configure viewport:
vp = Viewport(type = Viewport.Type.Ortho)
vp.camera_dir = (1, 1, -1)

# Configure renderer:
renderer = TachyonRenderer(shadows = False)    # hi-res
#renderer = OpenGLRenderer()    # lo-res

# Rendering loop:
for i,infile in enumerate(files):
outfile = "%s/%s.png" % (outdir,os.path.basename(infile))
print("%s --> %s" % (infile,outfile))

pipeline.source.load(infile)
vp.zoom_all()
vp.render_image(filename=outfile, size=(320, 240), renderer=renderer)