In the paper ‘On the typology and the worship status of sacred trees with a special reference to the Middle East’ Amot Dafni states:

“Gupta distinguishes a “tree-god”, whose worship became organized into a definite religion, from a “tree spirit”, whose propitiation degraded the level of sorcery and incarnation. In practice it is impossible to discern, “spirits”, “demons”, and “jinns” (general supernatural agents) as against “goddesses, “gods”, and “the deity” (religiously established worshipped elements). In the Middle East and North Africa, specific trees may be considered the abode of jinns, demons, or spirits, but these supernatural powers are never worshipped as a kind of “god”. No religious ceremonies are associated with or performed near these trees; these are regarded as heathen rites and are strictly prohibited.”

“Sacred groves were once widely protected from Africa to China [Gadgil 1991, Yu 1991]..throughout the Old World. They continued to be so protected even after conversion to Christianity in the tribal state of Mizorum in northeastern India, now being called ‘safety forest’, while the village woodlot from which regulated harvests are made are called ‘supply forest’ [Malhortra, 1990].” p281

[ http://ces.iisc.ernet.in/hpg/envis/mg/pdfs/mg138.pdf ]

So it would seem, that the notion of sanctuary is linked to the sacred and reserved value of the place/object.

“There’s nothing indefinite about the word ‘sanctuary.’It means safety. Protection was first offered, unspoken, by the topography of places where sanctuary was found: a hilltop, an island, a cave, a ring of encircling hills. Then came design: a shady grove, strong walls, or a single line of stone markers…

…Use divides sacred outdoor enclosures into public and private spaces. Public sanctuaries were where the oldest forms of worship were held. A community’s celebration of the gods in prayer, procession, and sacrifice was intended to order the rounds of nature. From drought, famine, plagues, war, and other disasters the sanctuary offered the hope of relief, if not relief itself. A sanctuary’s form often mirrored a culture’s concept of the cosmos, and its vision of an afterlife.”

See Mac Giswold ‘A History of the Sanctuary Garden’ published in Design Quarterly, June 22, 1996.

Perhaps the slave trade on Delos was considered to ‘pollute’ the site? There is reference in this document to two Bithynian slaves who were freed in the Delhic sanctuary in the middle of the 2nd century BC.
[ http://www.pontos.dk/publications/books/bss-6-files/bss6_15_avram ]