What concerns me is that you conducted this analysis in 3d-mode. For a true 2d-system like a graphene sheet, which is exactly planar in the undeformed configuration (I assume), the strain tensor calculation in 3d-mode will yield in unpredictable results --or an error message if you are lucky.

The reason is that you cannot calculate a 3x3 deformation gradient from a 2d-system where all atomic z-coordinates are zero. Tensor components such as F_zz, which involve the derivate of the displacement field with respect to the z-coordinate, remain undefined in this case. The implementation should catch this kind of degenerate situation, but it might not always be able to do so due to numerical errors. If that happens, tensor components like F_zz may take on arbitrary values. Consequently, the strain tensor E calculated from F will also be arbitrary (all of its components, not just the ones involving z-coordinates).

Have you tried doing this calculation in 2d-mode? You can switch to 2d-mode in the properties panel of the simulation cell. In 2d-mode, OVITO does not try to calculate the z-coordinate elements of the F tensor. Instead, they are set to fixed values assuming a plane strain situation.

If you can, please also post the simulation files for the initial and deformed configurations. Then I can take a look at the displacements of the atoms myself and check the strain calculation. I particularly would like to know if there are any out-of-plane displacements.