Few markets are are fast-paced, as competitive, as expensive, and as regulated as the New York City rental market. Apartments are scarce, and real estate agents often get bombarded with inquiries. Agents have limited time, so leads can fall through the cracks. Owners of brokerages, of course, and renters themselves, want all inquiries to be answered, and receive follow up, the kind of consistent follow up that builds a relationship and might eventually lead to a rental or even a sale. How could an AI assistant -- not a chatbot, but an autonomous agent operating via email and text with access to core databases and conversational context -- engage and nurture leads until they are to ready see an apartment or need advice from a licensed professional? This is exactly what the owner of a large NYC brokerage and I set out to discover.
We gave the assistant a name (let's call him Les), and a job description similar to a junior, unlicensed assistant. Les has a list of potential renters who need follow up. For each person on that list, he can only do a certain set of things, among them: look up a renter's file, search listings, send a message (via text or email), hand off to a human agent. Like any unlicensed person, he cannot negotiate lease terms, or answer questions based on opinion. And like a human agent he must never ever violate fair housing rules. Most important, we'd need to be able to see everything Les was doing: not just each conversation but the context behind it. And we'd need to measure performance: how many conversation, listings shared, agent handoffs and (ultimately) apartments rented.
On a technical level, Les takes shape not just through system prompts and API calls but through guardrails, in this case. In this case, an LLM evaluates each incoming and outgoing message for fair housing and unlicensed entity violations, automatically blocking responses that fail the test. A set of evals makes sure that Les handles a variety of situations with safety and accuracy, covering both "happy paths" (sharing listings that match the user's criteria), responses to "protected category" questions, and even responses to "abusive" language (Les, like any sane person, has a "4 strikes and you're out" policy). A dashboard lets admins view each conversational turn (including reasoning and tool calls), tracks LLM costs on a daily basis, and displays activity and success metrics.
I tested Les on my own (fantasy) search for a 2-bedroom apartment in the West Village (or Lower East Side) for me and my husband. Les was engaging, quick to respond, and consistently upbeat. He gave me a list of 2-bedrooms in my budget. When asked for more details, he supplied a virtual tour link and availability date. He sent me more listings with virtual tours so I could really see those apartments, and my life in them. He offered to connect me with an agent for an in-person tour. When new listings came on the market that met my criteria, he reached out. I was thrilled. Les was also able to handle the broker's more terse communications with ease. This week Les will meet real New Yorkers in the wild, but on a short leash. We'll see how it goes.