The inhomogeneous gravity field of an asteroid strongly perturbs nearby spacecraft trajectories. Orbits that would be bounded under Keplerian assumptions may instead evolve into collision or escape paths. This work addresses the challenge of designing safe and optimal control strategies for low-altitude orbits around asteroids. Two solution approaches are compared: direct transcription of the optimal control problem, and the soft actor-critic algorithm, a deep reinforcement learning method. Direct transcription boils down to a non-linear program which proves to be efficient in terms of delta-v consumption but becomes infeasible for certain test cases. In contrast, the soft actor-critic method produces a global policy that succeeds across the entire test set and offers faster evaluation. In both approaches, an additional penalty term is introduced to keep the spacecraft trajectory within a safe orbital shell.