Were all the townspeople devil worshippers as Young Goodman Brown thought?

This is not clear. We never know if Young Goodman Brown
actually experienced all of this or if it was a dream. Hawthorne did this deliberately.
The story is about the duality of humanity. We all, or all the townspeople, are capable
of good and evil. This is not to point out that we all are hypocrites. The point is that
good people are capable of bad deeds and vice versa.

The
other point of the story is that a failure to accept this is a denial of the reality of
the world. Goodman Brown believes he has lost his religious faith and his faith in
humanity at the very thought that a seemingly pious person, like Goody Cloyse, could
speak, let alone worship the devil. Brown thinks that if evil exists to that extent,
then there is no goodness in the world powerful enough to combat
it.

To rephrase your question in a metaphorical sense, I’d
ask if all the townspeople had evil thoughts, were tempted by evil deeds or actually
committed evil things. The answer is yes, but everyone to greater or lesser degrees.
Every human faces this duality. Everyone in the town has “walked with the devil at
night,” which is to say that even if someone chooses the righteous path, he/she must
have also contemplated the evil path. This is just the logic of choice. You can’t make a
choice unless you are presented with two or more options. This is the good/bad duality
characteristic of human nature.

Even Goodman Brown was left
wondering if it was a dream or not. In the end, it doesn't matter. Whether it was a
dream or a real experience, Brown could not accept the thought of such
evil.