Author Topic: histogram over all frames  (Read 69 times)

Kevin

  • Newbie
  • *
  • Posts: 19
histogram over all frames
« on: January 30, 2018, 11:07:19 PM »
Hello Dr.Stukowski,

I wrote a python script that can run with 'ovitos' command. The script reads dump file with multiple frames, and selects all the atoms that are classified as BCC (structure type 3). Then histogram data (BCC atoms counts with respect to Position in Z direction) is outputted. This histogram data is appended to the same output file for all the frames.

However, this script becomes slow after several frames, and even much slower as more frame is processed.
It takes a few seconds to proceed from frame 0 to frame 10, but it takes an hour to proceed to frame 100.

Can you please help me with this issue? Original code and sample output file are attached.


Code: [Select]
# Settings                                                                                         
outFilename='ovito_zprofile'
wchProfile='Position.Z'
number_bin=50
range_end=100

from ovito.io import *
from ovito import dataset
from ovito.modifiers import *
import numpy


node = import_file("dump.*",
                   multiple_frames = True)
fout = open(outFilename, 'wb')


for frame in range (node.source.num_frames):

    dataset.anim.current_frame = frame

    print(dataset.anim.current_frame)
    # Modifier 1: CNA                                                                             
    node.modifiers.append(CommonNeighborAnalysisModifier
                          (mode=CommonNeighborAnalysisModifier.Mode.FixedCutoff,
                           cutoff=3.5))
    node.compute()

    # Modifier 2: Selection                                                                       
    node.modifiers.append(SelectExpressionModifier
                          (expression = 'StructureType==3'))
    node.compute()
    # Modifier 3: Histogram                                                                       
    hist_modifier = HistogramModifier(bin_count = number_bin,
                                      particle_property = wchProfile,
                                      only_selected = True,
                                      fix_xrange = True,
                                      xrange_start = 0,
                                      xrange_end = range_end)
    node.modifiers.append(hist_modifier)
    node.compute()

    # Write                                                                                       
    header = ('timeframe = ' + str(dataset.anim.current_frame) + '\n' +
              'z\tcount(bcc atoms)\n')
    fout.write(header.encode('ascii'))

    numpy.savetxt(fout,
                  hist_modifier.histogram,
                  fmt='%d',
                  delimiter='\t',
                  comments='',
                  footer='\n')

fout.close()


Thanks,
Kyu

Alexander Stukowski

  • Administrator
  • Sr. Member
  • *****
  • Posts: 284
Re: histogram over all frames
« Reply #1 on: January 31, 2018, 01:23:35 PM »
Dear Kyu,

you made the classic mistake of appending modifiers to the data pipeline more than once from inside the for-loop. Note that every call to node.modifiers.append() extends the data pipeline by one step. Since you do this repeatedly, the pipeline will become longer and longer with every iteration. That is, it will contain the same modifiers several times. Thus, the CNA, selection and histogram computations will be performed multiple times for a single simulation frame when you call node.compute() to evaluate the pipeline.

The solution is to populate the pipeline with modifiers only once, i.e. move all calls to node.modifiers.append() in front of the for-loop (initialization phase). Then call node.compute() repeatedly inside the loop to evaluate the pipeline at different simulation times.

In a more recent version of the user manual I tried to make this pitfall more clear, because new users tend to make this kind of mistake. You can find the updated passage here (note that this is for Ovito 3.0.0, so some Python class names have changed):

http://ovito.org/manual_testing/python/introduction/modifiers.html#processing-of-time-dependent-data-and-simulation-trajectories


Kevin

  • Newbie
  • *
  • Posts: 19
Re: histogram over all frames
« Reply #2 on: January 31, 2018, 06:09:15 PM »
Thank you for the reply.

I modified the code as you proposed. I commented out all the node.compute() and only used node.compute() inside the for loop.
However, ovitos gives the following error:

current frame is =>  0  (=>this is output. Not part of the error message)
Traceback (most recent call last):
  File "script.py", line 41, in <module>
    node.compute()
  File "/Applications/Ovito.app/Contents/MacOS/Ovito.App/Contents/MacOS/../Resources/python/ovito/__init__.py", line 202, in _ObjectNode_compute
    if not self.wait(time = time):
  File "/Applications/Ovito.app/Contents/MacOS/Ovito.App/Contents/MacOS/../Resources/python/ovito/__init__.py", line 173, in _ObjectNode_wait
    raise RuntimeError("Data pipeline evaluation failed with the following error: %s" % state.status.text)
RuntimeError: Data pipeline evaluation failed with the following error: Cannot handle re-entrant modifier calls.
ERROR: The Python script 'script.py' has exited with an error.

Can you explain what "re-entrant modifier calls" is?
I also noticed weird behavior while debugging this code. If I use "node.compute(2)", then this code works, writing the histogram data on frame 2 over and over again, but it doesn't work for all other frames.


Code: [Select]
# Settings                                                                                 
outFilename='ovito_zprofile'
wchProfile='Position.Z'
number_bin=50
range_end=100

from ovito.io import *
from ovito import dataset
from ovito.modifiers import *
import numpy

node = import_file("dump.*",
                   multiple_frames = True)
fout = open(outFilename, 'wb')

# Modifier 1: CNA                                                                         
node.modifiers.append(CommonNeighborAnalysisModifier
                      (mode=CommonNeighborAnalysisModifier.Mode.FixedCutoff,
                       cutoff=3.5))
#node.compute()                                                                           

# Modifier 2: Selection                                                                   
node.modifiers.append(SelectExpressionModifier
                      (expression = 'StructureType==3'))
#node.compute()                                                                           
# Modifier 3: Histogram                                                                   
hist_modifier = HistogramModifier(bin_count = number_bin,
                                  particle_property = wchProfile,
                                  only_selected = True,
                                  fix_xrange = True,
                                  xrange_start = 0,
                                  xrange_end = range_end)
node.modifiers.append(hist_modifier)
#node.compute()                                                                           

for frame in range (node.source.num_frames):

    dataset.anim.current_frame = frame
    print('current frame is => ', dataset.anim.current_frame)

    node.compute()

    # Write                                                                               
    header = ('timeframe = ' + str(dataset.anim.current_frame) + '\n' +
              'z\tcount(bcc atoms)\n')
    fout.write(header.encode('ascii'))

    numpy.savetxt(fout,
                  hist_modifier.histogram,
                  fmt='%d',
                  delimiter='\t',
                  comments='',
                  footer='\n')

fout.close()


-Kyu
« Last Edit: January 31, 2018, 06:16:54 PM by Kevin »

Alexander Stukowski

  • Administrator
  • Sr. Member
  • *****
  • Posts: 284
Re: histogram over all frames
« Reply #3 on: January 31, 2018, 11:06:41 PM »
Your script looks okay. So I tried it on my computer and ran it with OVITO 2.9.0. I get the same error message.

The error must be caused by some unfavorable interplay between the SelectExpressionModifier and the HistogramModifier. If I take one of them out, the error disappears (but the script doesn't work as intended anymore, of course). I guess you discovered a bug in OVITO 2.9.0.

The good news is that the same script works normally when I run it using OVITO 3.0.0. Here, the pipeline system has been completely rewritten. So the bug is probably gone. My suggestion is that you switch to this newer version of OVITO, at least for this particular problem.

 

Kevin

  • Newbie
  • *
  • Posts: 19
Re: histogram over all frames
« Reply #4 on: February 01, 2018, 01:46:31 AM »
Thanks again for the reply.

I noticed the following behavior with node.compute() that may be relevant to current problem. I am wondering if this is a bug or if I am not correctly understanding the concept of OVITO architecture.

If node.compute() is used only once after setting up all modifiers (CNA, SelectExpressionModifier, Histogram), then the following error is invoked.

Error Message:
Code: [Select]
Traceback (most recent call last):
  File "script.py", line 38, in <module>
    node.compute()
  File "/Applications/Ovito.app/Contents/MacOS/Ovito.App/Contents/MacOS/../Resources/python/ovito/__init__.py", line 202, in _ObjectNode_compute
    if not self.wait(time = time):
  File "/Applications/Ovito.app/Contents/MacOS/Ovito.App/Contents/MacOS/../Resources/python/ovito/__init__.py", line 173, in _ObjectNode_wait
    raise RuntimeError("Data pipeline evaluation failed with the following error: %s" % state.status.text)
RuntimeError: Data pipeline evaluation failed with the following error: Cannot handle re-entrant modifier calls.
ERROR: The Python script 'script.py' has exited with an error.

However, if additional node.compute() is used after "SelectExpressionModifier", no error occurs and code works fine.
Is this behavior still due to a bug in Ovito 2.9.0 or is node.compute() necessary before Histogram is used?

I am wondering if the following quoted rule has anything to do with this behavior.
Quote
But now, since we want to directly access the pipeline results, we have to explicitly request an evaluation of the modification pipeline.
The quote is from here : https://ovito.org/manual/python/introduction/overview.html


Code:
Code: [Select]
# Settings                                                                                                 
outFilename='ovito_zprofile'
wchProfile='Position.Z'
number_bin=50
range_end=100

from ovito.io import *
from ovito import dataset
from ovito.modifiers import *
import numpy

node = import_file("dump.*",
                   multiple_frames = True)
fout = open(outFilename, 'wb')

# Modifier 1: CNA                                                                                           
node.modifiers.append(CommonNeighborAnalysisModifier
                      (mode=CommonNeighborAnalysisModifier.Mode.FixedCutoff,
                       cutoff=3.5))

# Modifier 2: Selection                                                                                     
node.modifiers.append(SelectExpressionModifier
                      (expression = 'StructureType==3'))
######                                                                                                     
#NOTE#: Without node.compute() below, error is invoked!                                                     
######                                                                                                     
#node.compute()                                                                                             

# Modifier 3: Histogram                                                                                     
hist_modifier = HistogramModifier(bin_count = number_bin,
                                  particle_property = wchProfile,
                                  only_selected = True,
                                  fix_xrange = True,
                                  xrange_start = 0,
                                  xrange_end = range_end)
node.modifiers.append(hist_modifier)
node.compute()

header = ('timeframe = ' + str(dataset.anim.current_frame) + '\n' +
              'z\tcount(bcc atoms)\n')
fout.write(header.encode('ascii'))

numpy.savetxt(fout,
              hist_modifier.histogram,
              fmt='%d',
              delimiter='\t',
              comments='',
              footer='\n')
fout.close()

« Last Edit: February 01, 2018, 01:49:35 AM by Kevin »

Alexander Stukowski

  • Administrator
  • Sr. Member
  • *****
  • Posts: 284
Re: histogram over all frames
« Reply #5 on: February 01, 2018, 10:35:56 AM »
The data pipeline consists of the sequence of modifiers that you add using the modifiers.append() method. The compute() method requests an evaluation of the data pipeline. In other words, it lets every modifier in the sequence act on the atomic data, one after the other. The result of these computations is returned by compute() itself or can be accessed later on through the node.output field. If you call compute() before the HistogramModifier has been added to the pipeline, it will compute the results only up to and including the SelectExpressionModifier. I guess so far everything has been clear to you already.

Behind the scenes, however, a caching mechanisms is in place. The details of this mechanism have changed from OVITO 2.9.0 to 3.0.0, but in the old version it worked as follows: After a pipeline evaluation is complete, OVITO stores the resulting data in the node.output field. In addition it memorizes the point up the which the pipeline was computed, in this case the SelectExpressionModifier. And it memorizes the animation time for which it was computed (dataset.anim.current_frame).

If you add the HistogramModifier to the pipeline after the first call to compute() and then call compute() a second time, OVITO will remember that the data currently in the cache already represents the state after the SelectExpressionModifier. Since the pipeline has only changed behind this caching point but not in front of it, OVITO can directly continue with the application of the HistogramModifier to the cached data. No re-evaluation of the other two modifiers takes place. I think this is why the error doesn't occur in this case. The error seems to be caused by the successive evaluation of the SelectExpressionModifier and the HistogramModifier.

This would also explain why the error occurs again once you change the current animation time (dataset.anim.current_frame): OVITO recognizes that the cached data is out of date and a complete pipepline evaluation is triggered, leading to the error.
« Last Edit: February 01, 2018, 10:43:07 AM by Alexander Stukowski »

Kevin

  • Newbie
  • *
  • Posts: 19
Re: histogram over all frames
« Reply #6 on: February 05, 2018, 06:20:51 PM »
1st node.compute() right after CNA modifier, and 2nd node.compute() after histogram modifier also does not trigger the error.

Since this issue is resolved in 3.0.0 version of OVITO, it may not be important issue, but I just wanted to refer this issue in case this matters to the developers.

Thanks again for the help.