a burr in/under the saddle

someone or something that is a constant cause of trouble or annoyance