Identifying a general quasilocal notion of energy-momentum and angular momentum would be an important advance in general relativity with potentially important consequences for mathematical and astrophysical studies in general relativity. In this paper, we study a promising approach to this problem first proposed by Wang and Yau in 2009 based on isometric embeddings of closed surfaces in Minkowski space. We study the properties of the Wang-Yau quasilocal mass in high-accuracy numerical simulations of the head-on collisions of two nonspinning black holes within full general relativity. We discuss the behavior of the Wang-Yau quasilocal mass on constant expansion surfaces, and we compare its behavior with the irreducible mass. We investigate the time evolution of the Wang-Yau quasilocal mass in numerical examples. In addition, we discuss mathematical subtleties in defining the Wang-Yau mass for marginally trapped surfaces.