I know the rule: a collective noun may be used with a singular or plural verb, but what about when the collective noun is this kind of compound one -
'A coven of self-styled witches'
Do I follow this with 'are' or with 'is'? If I had just used 'coven' I would follow it with 'is' to emphasise the unity of the group; if I just used 'witches' I would follow it with 'are', but I don't know which to choose for the compound phrase.

I would always have used a collective noun with a singular verb but I'm no expert so I can't give you a definitive answer. Instead I'll add another question.
What about 'a lot'? Is that with singular or plural? I'm not sure. For example I might say 'There are a lot of theories that...' or 'There's a lot of people...' but yet I would say 'A lot of people think...' and not 'A lot of people thinks...'. Anybody know what it should be?

I believe you should say

"A coven of self-styled witches is..."

because you're still talking about them as a unit. As with:

The flock of birds was making its way across the sky.

The flock of sheep is stationery in the field.

Don't be fooled by the witches, birds and sheep which in these sentences act as descriptions of the coven or flocks.

Deb

What about 'a lot'?

I don't think "a lot" is a collective noun. I don't know how it came about but would consider "a lot of" to have the same meaning as and be interchangeable in a sentence with "many".

Hence

'There are many theories that...'

'There are many people...'

'Many people think...'

Thus, if you want to use "a lot", which I don't personally find particularly elegant, I'd write:

There are a lot of theories that...

There are a lot of people...

A lot of people think...

Deb