Last night, Russian punks Pussy Riot went toe-to-toe with Stephen Colbert on his show The Colbert Report, showing a wry sense of humor about their unjust imprisonment at the hands of President Vladimir Putin. Toward the end of the interview, they also invited Colbert to become a member of Pussy Riot — provided he talk to his conservative “friend” Putin.

Along with their translator, Colbert spoke to Nadya Tolokonnikova and Masha Alekhina for a little over 12 minutes, focusing largely on Putin’s Russia and their roles as the country’s most visible protesters. Colbert, of course, played his character, but Tolokonnikova and Alekhina were more than happy to volley back at him. They even got him to crack a smile when they responded to his joke about being searched upon leaving the studio with, “We’ve had two years of practice hiding things from searches.”

In criticizing Putin they were harsh but humorous, noting that if Putin released them from prison for good publicity before the Winter Olympics in Sochi, it hasn't actually helped the country’s image. “Maybe Putin made a mistake and should throw us back in jail,” they said. But their most succinct criticism of Putin was also their funniest: “We don’t want a shirtless man on a horse leading us around.”