Modulation theory, based on a Lagrangian formulation of the governing equations, is used to investigate the propagation of a nonlinear, nonlocal optical vortex solitary wave in a finite nematic-liquid-crystal cell. The nematic response to the vortex is calculated using the approach of themethod of images (MOI). It is demonstrated that the MOI is a reliable alternative to the usual Fourier series solution as it requires an order of magnitude fewer terms to obtain excellent agreement with numerical solutions. It is found that the cell walls, in addition to repelling the optical vortex solitary wave, as for an optical solitary wave, can destabilize it due to the fixed director orientation at the walls. A linearized stability analysis is used to explain and analyze this instability. In particular, the minimum distance of approach of a stable vortex to the wall is determined from the stability analysis. Good agreement is found with numerical minimum approach distances.