Should I avoid this family?

I have been seeing/ visiting my neighbors for about a year and a half now. I recently stopped (October to be exact) and I haven't visited them since. I just didn't feel the connection anymore, and the grandma there was too religious for my taste and a real sour puss. I used to like that, but I grew up a bit and realized I wasn't into that. Just today, my mom drove me over there without telling me, and the family smiled at me while the grandma bombarded me with questions as to why I wasn't visiting. And then when we were about to leave, they told me to visit them, or else they'd drag me to their home. My mom likes these people. A lot. But there's no "real" communication between them. When I did visit them a lot, I was overly cheerful and go-lucky naturally. Now I'm calm, thoughtful, and I prefer to be quiet and to myself. How do I deal with this?