Hermits were individuals who lived on their own.
The Fourth Doctor told Romana II that he knew an old hermit who lived in the misty mountains of Southern Gallifrey, who told him of the Record of Rassilon. (TV: State of Decay) The Doctor once told Jo Grant about his home: "When I was a little boy, we used to live in a house that was perched halfway up the top of a mountain. Behind our house, there sat under a tree an old man. A hermit, a monk. He'd lived under this tree for half his lifetime, so they said, and had learned the secret of life. So, when my black day came, I went and asked him to help me." (TV: The Time Monster) The young First Doctor would play truant so that he could visit him. (PROSE: The Eight Doctors)
The Sixth Doctor, confused by his fresh regeneration, decided to become a hermit and bring his companion Peri Brown with him, as his disciple and servant. The asteroid Titan III was the supposed hermitage. Soon, he cleared up his mind. (TV: The Twin Dilemma)
A member of the Celestial Intervention Agency who later joined the Celestis as an Investigator became a hermit in Mictlan. (PROSE: The Taking of Planet 5)
The Tenth Doctor told Professor Yana that he was a "bit of a hermit." Pointing out Martha and Jack, Yana replied, "A hermit... with friends?" to which the Doctor replied that they were "Hermits United" who met up every ten years to discuss caves. (TV: Utopia)