Yes. One should realise, the potential enemy was under fire of arrows and stones when trying to conquer the ditch and tall fence. Furthermore, a fence, preferably constructed of freshly cut plants, offers a very good protection against fire. However, against a large and modern army of the high- or late Middle Ages, neither of those solutions offered any protection. Most castles of the lower nobility were not meant for that, they protected against wild animals and neighbours of lower nobility, eager with feud.