Building a Better Back Off: Interview with Eloquent intern Justin Dieter

Eloquent’s head of product, Josh Issler, sat down with Eloquent’s summer intern, Justin Dieter, to discuss the work he did over the summer building a smart back off for Eloquent’s chatbot.

Hi Justin! Why don’t you introduce yourself?

I’m from Colorado, just recently started my sophomore year. I came into Stanford really interested in AI. During my freshman year, I took as many AI classes as I could so I could do research in my areas of interest: NLP, Reinforcement Learning, and Computer Vision.

What have you been working on at Eloquent?

I’ve been working on an NLP model that helps the bot more gracefully back off when it doesn’t understand what a customer says. This means the bot will be able to respond intelligently to requests or questions it’s not able to answer.

Before my model, if a customer says something our bot isn’t sure how to respond to like “When is my birthday?”, we would respond by saying something like, “I’m sorry, I must have missed something in there.” That’s a frustrating response and can really make it feel like the customer isn’t being fully listened to.

Now, with my model, if someone says “When is my birthday?”, our bot can respond by saying “I do not know when your birthday is.” Although the bot still doesn’t understand what the customer said, it backs off by specifically referencing the customer’s words. It makes you feel more listened to and is a much nicer customer experience.

Justin’s model helps Elle rephrase the confusing utterance.

What was the most difficult problem you faced while building your model?

Collecting data specific to the task. There’s no pre-existing dataset for this, so I had to build one myself. I used Amazon Turk crowd-workers to get my dataset, but it’s extremely difficult to frame the instructions just the right way. You want the crowd-workers to correctly answer your prompt, of course. But they need to answer in a way that doesn’t show too much creativity, so a neural model can learn the correct patterns.

How’d you handle that?

I gradually iterated on instructions for the task. By being extremely specific in the details of what the crowd-workers had to do, I created a set of instructions that allowed them to reliably produce data fit to the task.

Justin presenting at BayLearn 2018. Link to poster at the bottom of the interview.

How would you describe working at Eloquent?

Eloquent has a simultaneously laid back and urgent atmosphere. There’s always important work to be done, but the culture still has a calm and happy feel to it. It was a really fun place to intern because the work you do is important to the actual product. You get hands on experience and you feel valued, but you’re also not crazy stressed out.

Working for Eloquent has really improved my ability to do meaningful research. I came in with a very good theoretical understanding of AI, but building something to be actually used in the product definitely sharpened my skills and improved the speed at which I can build models and collect data. Access to experienced researchers at Eloquent also really helped me learn quickly. After my time at Eloquent, I would consider myself a much more competent researcher.

What’s next for you?

I still have more work to do in order to publish a paper for the research I did at Eloquent. Outside of school, that’s my top priority. After that, I have lots of ideas I hope to try out. I want to produce more exciting research!

Best of luck to you Justin, from everyone at Eloquent!

Link to Justin’s poster: “I don’t know what to title this poster”

If you have any questions or comments for Justin, please email them to [email protected] and they will be forwarded to Justin’s personal email. Thanks for reading!