"Don’t ever take a fence down until you know the reason why it was put up,"

Actually, the post title is not a direct quote of Chesterton, but a paraphrase of his words (attributed to GKC by John F. Kennedy). Here is the passage JFK was paraphrasing:
_____________________

In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, "I don't see the use of this; let us clear it away." To which the more intelligent type of reformer will do well to answer: "If you don't see the use of it, I certainly won't let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it."

This paradox rests on the most elementary common sense. The gate or fence did not grow there. It was not set up by somnambulists who built it in their sleep. It is highly improbable that it was put there by escaped lunatics who were for some reason loose in the street. Some person had some reason for thinking it would be a good thing for somebody. And until we know what the reason was, we really cannot judge whether the reason was reasonable. It is extremely probable that we have overlooked some whole aspect of the question, if something set up by human beings like ourselves seems to be entirely meaningless and mysterious. There are reformers who get over this difficulty by assuming that all their fathers were fools; but if that be so, we can only say that folly appears to be a hereditary disease. But the truth is that nobody has any business to destroy a social institution until he has really seen it as an historical institution. If he knows how it arose, and what purposes it was supposed to serve, he may really be able to say that they were bad purposes, or that they have since become bad purposes, or that they are purposes which are no longer served. But if he simply stares at the thing as a senseless monstrosity that has somehow sprung up in his path, it is he and not the traditionalist who is suffering from an illusion. We might even say that he is seeing things in a nightmare. This principle applies to a thousand things, to trifles as well as true institutions, to convention as well as to conviction. It was exactly the sort of person, like Joan of Arc, who did know why women wore skirts, who was most justified in not wearing one; it was exactly the sort of person, like St. Francis, who did sympathise with the feast and the fireside, who was most entitled to become a beggar on the open road. And when, in the general emancipation of modern society, the Duchess says she does not see why she shouldn't play leapfrog, or the Dean declares that he sees no valid canonical reason why he should not stand on his head, we may say to these persons with patient benevolence: "Defer, therefore, the operation you contemplate until you have realised by ripe reflection what principle or prejudice you are violating. Then play leapfrog and stand on your head and the Lord be with you."

-The Thing: Why I am a Catholic (1929)