Yeah. Did that once. Which was odd. I went to an office in SF to talk on a videoconference with interviewers in New York. Strange.
As long as you don't stand up, you don't have to wear pants. Though I would recommend that you don't, as you never know when you'll get a better opportunity to tell them you're not interested in the position.