Firstly, congratulations – I think it’s getting progressively harder to get published so it’s great that you have an offer. However, I think you’re wise to consider carefully whether this is right for you.
With a small publisher, it’s really difficult to get your book into bookshops so, if that was your main rationale for wanting a physical book, you might be equally disappointed even with a bigger small publisher. I suppose it depends on your overall view of ebooks – some people prefer them, others don’t like them. And, even if you don’t manage to get your books into a bookshop, there’s still the pleasure of signing with the physical book.
I’ve written a few articles about my experiences negotiating with a small publisher without an agent – the latest is here
https://thecontemporarysmallpress.com/2016/03/02/riding-the-mule-the-adventure-of-small-press-publication/
I’d say try to be clear about what you want and negotiate on this basis rather than assuming you have to take the whole package as offered.
Good luck – I appreciate that you and your agent might now have settled this between you but thought it wouldn’t harm to add my perspective.