A compliance expert quizzed ChatGPT. Here’s what it got wrong.

The results? Sometimes it got part of the answer right. Other times, it was flat-out wrong or lacked depth. Below you’ll find the questions, its answers and my rebuttals as a compliance consultant.

Why did I do this? Because everyone should be doing it — testing artificial intelligence (AI), that is. This little “quiz” is more than a parlor trick on paper. My ulterior motive is to wake you up and push you toward the edge, where you can see the 360-degree view of today’s tech landscape. This perspective will remind you to use AI with care, not in haste.

We’re living in a moment where AI is framed as everything, from the cure-all to the magic solution, even “godlike,” to quote Brad Inman. AI is formidable and can seemingly do it all, sometimes in the most amazing ways. But amazing isn’t perfect, and confusing the two can be dangerous.

Even at the risk of contributing to the AI fatigue many of us are feeling, with every headline, LinkedIn post and industry panel tethered to the topic these days, I’ll still raise my hand and say what’s on my mind: Even though AI has already been crowned the all-powerful wizard, I’ve been reflecting on a more nostalgic one from my childhood, a wizard whose lessons still hold timeless truths.

I was watching The Wizard of Oz with my son the other night when this idea bloomed in Technicolor. In the classic film, Dorothy and her friends discover that the mighty and mysterious Wizard isn’t so mighty after all — just a man behind a curtain, pulling levers and projecting illusions.

In the same way, the Wizard’s illusion reminds us that engaging AI isn’t just “push a button, get the truth.” You have to apply your mind, your real-world experience and your know-how. Or, to borrow traits from the memorable cast: Use your brain, bring your heart and carry your courage. (OK, that last part might be a little dramatic.)

Put simply, you have to vet and challenge AI the same way you’d verify any other source of information. And if you don’t? Well, that risk had better be one you’re willing to take. AI may help solve tough questions, but it can just as easily create problems if you rely on it blindly. I’m already seeing that misstep in my neck of the woods, with agents leaning too heavily on AI’s analytical tools.

I may sound skeptical, but trust me, I’m not. I see its tremendous value, and I’m on board the AI train. Still, I have to shake the tree once in a while, because that’s what compliance consultants do, no matter the audience or the inconvenience it might cause.

‘Letter of the law’ quiz (Part 1)

Q1. Which law allows a California real estate broker to appoint a DRE-licensed branch or division manager, and what conditions would disqualify an appointee?

• Have had their license suspended, restricted, or revoked; or

• Are subject to disciplinary action that prohibits such appointment.

Summer: Not completely correct. The appointee can also be a salesperson with two years of full-time real estate experience within the five years preceding the appointment. In addition, the law expressly disqualifies anyone who is or has been subject to an order of debarment.

Q2. What are the disclosure rules when a real estate broker also arranges the financing in a transaction? Which legal code sections apply?

ChatGPT Answer: A broker who negotiates both the sale and financing must be properly licensed and, if arranging loans, hold an MLO endorsement (NMLS). Key requirements include: written disclosures, RESPA compliance, and avoiding conflicts of interest.Citations: Bus. & Prof. Code §10131(d), §10131.1, and Commissioner’s Regulation 2840 et seq.

And here’s the important part: Even a seemingly “small” mistake — like depositing in three days instead of one — is still a trust fund violation, and the DRE takes those violations very seriously.

Summer: Wrong code section. The correct citation is Bus. & Prof. Code §10133(a)(1), and the law does not specifically reference LLCs, only corporations or general partnerships. Whether and how this applies to LLCs is a more nuanced discussion for another day and one I won’t unpack fully here.

Summer: To be clear, the law requires a broker to report misconduct to the Real Estate Commissioner when a licensee is terminated for a violation (e.g., misrepresentation) that constitutes grounds for disciplinary action under the Real Estate Law. Failing to report could be extremely problematic and may even be deemed guilty knowledge of the agent’s misconduct. The applicable statutes are Bus. & Prof. Code §§10178 and 10179. The DRE has even labeled this reporting duty part of its “Benefit of the Doubt” program.

Summer: Although you’re not wrong, this is where experience makes all the difference. I once saw a broker face formal discipline over what began as a “small” advertising violation. A salesperson had run advertising missing DRE-required disclosures — not unusual, and not something you’d expect to trigger a major investigation.

What looked like a technical slip in advertising unraveled into much more. This is why I often say compliant advertising can act as a shield to regulatory scrutiny. Either way, it’s the kind of story you won’t find in a law book. It comes from on-the-ground experience, compliance work in the trenches and real life.

ChatGPT Answer: A transaction coordinator (TC) licensed under another broker cannot perform licensed activity for a different broker. If they assist with transaction review, it may constitute unlicensed activity (since they’re not affiliated under the broker’s license).

Summer: This is a tricky question — you’re not entirely wrong, but there’s a lot more to it.

First off, if we are strictly talking about transaction coordination, then it does not require a real estate license in California. In practice, many TCs are licensed with one brokerage but perform coordination for others.

The real problem arises when brokers delegate supervisory work to a TC who isn’t licensed under their brokerage. That’s where non-compliance becomes inevitable.

You cited one code section correctly (though it doesn’t always apply). The fuller picture involves Bus. & Prof. Code §§10130, 10159.2, 10177(h) and Commissioner’s Regulation 2725.

After I gave ChatGPT my rebuttals, I asked if it agreed. Here’s what happened:

On the statute questions, it conceded that my corrections added missing statutory precision, fixed mis-citations, and clarified enforcement nuances it had glossed over.

On the experience questions, it essentially admitted the obvious: It doesn’t know what it can’t know. These situations live outside the four corners of a statute and, often, beyond the reach of the web. They demand judgment, practical wisdom and real-world experience.

Even though I knew the answer, I asked ChatGPT if it “learned” through this exercise. It replied: “I can incorporate your corrections in this conversation and explain the statutes more precisely, but I don’t replace professional judgment. Compliance lives in context, and people still matter.”

The goal of this piece isn’t to spotlight what I know, but to illustrate what AI might not (at least not today). Ultimately, it’s the user behind the screen who must think critically before relying on AI for crucial information.

So where did the yellow brick road take us? If you’re going to use AI, don’t forfeit your own smarts, experience and informed questions. Even when AI’s answers are technically correct, they often stop short of the full picture; in compliance, overlooked details can make all the difference.

You, the user, play an integral role in shaping the outcomes AI produces. Call that a human feature, not a flaw. Don’t just receive; look behind the curtain and vet the illusion of perfection. Use AI responsibly, test and challenge its outputs, and be intentional about how you harness its power.

Just ask the Scarecrow, the Tin Man and the Lion. A brain, a heart and courage aren’t in the code. They aren’t in AI either. They live in the flesh. The wizard of compliance still needs a human to hold the reins. In the end, AI’s reputation won’t endure — yours will.

NOTE: The opinions and recommendations expressed in this article are based on Summer Goralik’s experience as a real estate compliance consultant and former investigator for the California Department of Real Estate. They are provided for informational purposes only and should not be construed as legal advice. Readers should consult with their brokerage and/or qualified legal counsel in their jurisdiction for guidance on specific situations.

About Summer