Baby was the name the Eleventh Doctor gave to the language supposedly spoken by human babies who otherwise seemed as though they had not yet learned the capacity to speak. It was spoken by Alfie Owens (TV: Closing Time) and Melody Pond. (TV: A Good Man Goes to War) It appeared to be a one-way language, spoken by babies and heard by the auditors. The listener — in these cases, the Doctor — responded in standard English, which the babies seemed to understand.
The TARDIS translation matrix, however, did not translate Baby for other listeners. Only the Doctor appeared able to communicate with them in this way. Judging solely by the Doctor's responses, "Baby" was a complex language with a complete grammar, a large vocabulary and many rich colloquialisms. (TV: A Good Man Goes to War, Closing Time)
The Fourth Doctor also indicated an ability to speak Baby. When he and Sarah were in Pescaton-terrorised London, they came across an apparently abandoned baby. Unable to get any information from the child, the Doctor said, "He won't talk." Unaware that the Doctor may have spoken literally, Sarah responded, "He can't talk." (AUDIO: Doctor Who and the Pescatons)