Author Topic: Crash of ovitos (WS analyser) for very large number of atoms?  (Read 2742 times)

Xtof

  • Jr. Member
  • **
  • Posts: 66
Crash of ovitos (WS analyser) for very large number of atoms?
« on: October 04, 2017, 12:14:03 PM »
Hi all,

A few months ago I wrote a Python script to analyse defects that form during cascades (with the WS analyser). I used it many times and it works fine.
However, yesterday it crashed with the following error message:

ERROR: LAMMPS dump file parsing error. Invalid number of atoms in line 4:
1002514050

ERROR: The Python script 'loop_script_ovito_analysis_good_cascades.py' has exited with an error.
Traceback (most recent call last):
  File "loop_script_ovito_analysis_good_cascades.py", line 77, in <module>
    node = import_file(file_coord_final)
  File "/marconi/home/userexternal/cortiz00/ovito-2.8.0-x86_64/lib/ovito/plugins/python/ovito/io/__init__.py", line 117, in import_file
    if not node.wait(signalError = True):
  File "/marconi/home/userexternal/cortiz00/ovito-2.8.0-x86_64/lib/ovito/plugins/python/ovito/__init__.py", line 160, in _ObjectNode_wait
    raise RuntimeError("Data pipeline evaluation failed with the following error: %s" % state.status.text)
RuntimeError: Data pipeline evaluation failed with the following error: LAMMPS dump file parsing error. Invalid number of atoms in line 4:
1002514050

Is it due to a too large number of atoms in the box (~ 1000 millions)? Is there a limit to the number of atoms in ovitos?


Many thanks in advance and best regards,
Christophe

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #1 on: October 04, 2017, 01:35:00 PM »
Hi Christophe,

There is no problem with your script. The current LAMMPS dump file reader still contains a check on the number of atoms which prevents loading files with more than 10^9 atoms. This sanity check was written 9 years ago, in times when multi-billion atom simulations were far less common (and visualising dataset of this size on a desktop computer virtually impossible anyway).

I just made changes to the code to push this limit to 2^31 atoms. This, however, is the current hard limit of OVITO's internal data model. It uses 32-bit values to store integer particle properties including the atom IDs. I will extend the data model in the coming weeks or months to further push the internal limit to 2^63 atoms or so, but this will require significant development work and testing. I acknowledge that analysing large multi-billion datasets will soon become a common practice, even before graphics hardware catches up to make the visualization possible.

Please stand by until I release a new dev build next week, or pull and build the updated sources from Gitlab yourself if you need to process your >1e9 dataset immediately.

-Alex 

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #2 on: October 04, 2017, 01:40:26 PM »
Hi Alex,

Thanks for the reply. I guessed it was due to this sort of limit. No problem, meanwhile, I will "remove" these cascades from the set to analyse.

I take the opportunity to ask you if there would be a way to accelerate this kind of analysis. To analyse 6 cascades (huge boxes), it took something like 7 hours. And I know ovitos is already very fast. Would you have any suggestion?

Best regards,
Christophe

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #3 on: October 04, 2017, 04:37:57 PM »
If 6 configurations take about 7 hours, this makes about 1 hour per dataset. How many atoms does your dataset comprise?

The WS analysis consists of four steps:

1. Loading the displaced configuration from disk
2. Loading the reference configuration from disk
3. Building the hierarchical nearest-point-search data structure
4. The query phase, where physical atoms are assigned to their nearest reference site

We'll have to do some profiling to find out how long these individual steps take. The first step can be timed by adding a print statement after the import_file() call in the Python script. Step 2 takes the same amount of time as step 1. For profiling steps 3 and 4 I'll have to amend the code a bit.

None of these steps is currently parallelized. Step 4 could be made parallel, but this would probably increase the memory footprint. Whether it is worth the development effort depends on the profiling. Steps 1 and 2 can be speed up considerably by using a binary file format such as NetCDF instead of text-based dump files.

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #4 on: October 05, 2017, 09:22:55 AM »
The number of atoms per dataset varies. It ranges from 89 million to 635 million atoms. In average, each dataset comprises 340 million atoms. The size of some files can reach 24 GB. I understand it can take some time to read and load so many data.

I will check how long it takes to load the displaced configuration file.

I know that LAMMPS allows dumping files as .bin. Would it make it? Would ovitos be able to read these files?

Christophe

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #5 on: October 05, 2017, 10:12:25 AM »
I have created an issue to track these two topic:

Support for >2 billion atoms:

https://gitlab.com/stuko/ovito/issues/51

Improving WS analysis performance:

https://gitlab.com/stuko/ovito/issues/50

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #6 on: October 05, 2017, 10:14:17 AM »
Yes, OVITO and ovitos can read binary LAMMPS dump files. But I recommend the NetCDF format as a much better alternative. See the dump netcdf command of LAMMPS. Ovito/ovitos can read this format, and soon will be able to write it as well.

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #7 on: October 06, 2017, 04:17:33 PM »
I have monitored the times for the different steps.

- Loading the displaced configuration from disk (635 million atoms, 15 GB): 199.38 s
- node.compute() with the WS analyser: 10507 s

I guess I could try the NetCDF format to speedup the loading of files, but clearly, most of the time is spent on the calculation of defects.

Best regards,
Christophe

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #8 on: October 07, 2017, 05:01:00 PM »
The Wigner-Seitz routine has been parallelized now and makes use of all processor cores. Please try the new test build dev38.

(And this one should also work for up to 2 billion atoms)
« Last Edit: October 07, 2017, 05:02:56 PM by Alexander Stukowski »

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #9 on: October 09, 2017, 09:35:09 AM »
Oh waouh. That was fast!
Thanks a lot, Alex. I am going to try it right away. I'll let you know.

Best regards,
Christophe

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #10 on: October 09, 2017, 09:37:33 AM »
Please wait a moment. I was notified yesterday that the dev38 build for Linux is broken (some missing libraries etc). I am fixing it at this moment.

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #11 on: October 09, 2017, 11:09:09 AM »
Version dev39 for Linux is now available for download. Please use this one for testing.

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #12 on: October 09, 2017, 01:58:53 PM »
I tried it on my PC (i7 quad) for boxes of moderate size.
Older version: 10min30s
dev39: 5min24s
Already a factor x2.

Now I am trying it on very large boxes (~635 M atoms) on a Xeon Phi (KNL) and on a 24 cores SkyLake. I will let you know.

Best regards,
Christophe

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #13 on: October 10, 2017, 09:15:59 AM »
I tried the new version dev39 on the HPC Marconi with 18 boxes with, on average, 635 M atoms.

Old version (serial) on a proc Intel SkyLake: 21h30min
New version dev39 with a proc Intel SkyLake (24 cores): 1h47 min
New version dev39 with an Intel Xeon Phi (Kight Landing 68 cores): 8h31min


Thank you, Alex and congratulations for the great work. This will clearly ease the analysis of defects in large boxes.

With best regards,
Christophe

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #14 on: October 20, 2017, 10:16:14 AM »
In the latest dev build of OVITO I have switched from single-precision to double-precision numbers as default floating-point data type. All calculations will now be performed with double precision in OVITO and the memory footprint becomes larger.

I would be interested in determining how that affects performance. If you have the time, it would be great if you could redo the benchmark run with the newest OVITO build to see if there is any measurable effect.

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #15 on: October 20, 2017, 10:23:23 AM »
Ok, no problem.

I will do the benchmark with the old version, the version 39 (WS parallelized) and the new version.

Best regards,
Christophe

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #16 on: October 20, 2017, 11:40:06 AM »
Here is the result. The benchmark was performed with a proc 24-cores Intel Xeon 8160 CPU (Skylake) at 2.10 GHz (HPC Marconi)

Analysis of 30 boxes of 200x200x200 (bcc Fe, 16 million atoms)
dev39: 6m28s
dev46: 6m23s

Analysis of 30 boxes of 300x300x400 (bcc Fe, 72 million atoms)
dev39: 33m28s
dev46: 33m23s

We can say that the CPU time is more or less the same. The performance is not affected by the double-precision.

If you need a reference, I can also do it with the old version of Ovito (WS not parallelized). Just let me know.

Best regards,
Christophe

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #17 on: October 20, 2017, 11:47:33 AM »
Great. Thanks for the quick effort.
Looks like the extra costs of double-precision computations are negligible. That's good news.

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #18 on: October 30, 2017, 07:15:13 PM »
Christophe,

I'm sorry, but I now noticed that there was a mistake. As it turned out, the dev46 version double precision computations were not enabled in the dev46 build. I didn't immediately notice it, because my Linux virtual machine only allows me to build the Ovito executable but not run it.

Not surprising then, of course, that your measurements didn't show any significant deviations from the dev39 build. I'm sorry for having wasted some of your time here. If you still have the chance, it would be nice if you could run the benchmark again with the dev52 build. It has double precision calculations enabled (see OpenGL information dialog, accessible from the help menu).

-Alex

Xtof

  • Jr. Member
  • **
  • Posts: 66
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #19 on: October 31, 2017, 09:21:26 AM »
No problem. I launched the analysis (Python script) of 30 boxes of 200x200x200 (bcc Fe, 16 million atoms) on a proc 24-cores Intel Xeon 8160 CPU (Skylake) at 2.10 GHz (HPC Marconi).

dev39: 6m28s
dev52: 6m28s

This time we can say that there is no extra cost in using double precision.

Christophe

Alexander Stukowski

  • Administrator
  • Hero Member
  • *****
  • Posts: 637
Re: Crash of ovitos (WS analyser) for very large number of atoms?
« Reply #20 on: November 01, 2017, 01:43:25 PM »
Okay, cool.

Thank you!