"Of course you can," he replied, more sincerely. "I was only joking. You can even actually come in, if you'd like. Glance at my books, if you want."
Other options:
no subject
"Of course you can," he replied, more sincerely. "I was only joking. You can even actually come in, if you'd like. Glance at my books, if you want."