Firstly, congratulations – I think it’s getting progressively harder to get published so it’s great that you have an offer. However, I think you’re wise to consider carefully whether this is right for you.
With a small publisher, it’s really difficult to get your book into bookshops so, if that was your main rationale for wanting a physical book, you might be equally disappointed even with a bigger small publisher. I suppose it depends on your overall view of ebooks – some people prefer them, others don’t like them. And, even if you don’t manage to get your books into a bookshop, there’s still the pleasure of signing with the physical book.
I’ve written a few articles about my experiences negotiating with a small publisher without an agent – the latest is here
I’d say try to be clear about what you want and negotiate on this basis rather than assuming you have to take the whole package as offered.
Good luck – I appreciate that you and your agent might now have settled this between you but thought it wouldn’t harm to add my perspective.