Interactive Marker with rosbag "ROS Time moved backwards"

I'm using rqt_bag to visualize a rosbag in RViz. Independently of that, I publish a interactive marker that changes colors based on a topic being published. This works well, until I move backwards in the rosbag, then the interactive marker throws an exception because the ROS time moved backwards. Specifically, the sleep() method in the rospy timer (http://docs.ros.org/jade/api/rospy/html/rospy.timer-pysrc.html). I've tried changing the interactive marker header stamp to rospy.time.now() and I've tried just a static non-changing time. I've also tried deleting the interactive marker whenever I go backwards in time, but the error procs before I can delete the marker. Am I trying to do something that is not possible with the way Interactive Markers work? I have use_sim_time set to true and --clock set. Thanks