When people say that to me I reply -"Oh so you think we're just animals."
There are only two life forms on this planet - plants & animals. Which group do you fall into?
For the sake of this argument we can ignore some lifeforms that defy taxonomy.
And just for the record - I'm rather happy to be an animal as I have no desire to be a plant.