Despite the recent identification of RFX4 as a neurodevelopmental disorder risk gene, its role in cortical development remained unclear. Here, we identified both shared and lineage-specific RFX4 requirements for human cortical development using new human stem cell models of deficiency and pathogenic mutation. We found that RFX4 restrains neurogenesis by acting cooperatively with NOTCH signaling, specifically repressing pro-neuronal and synaptic gene expression in neural progenitors. We also determined that genome-wide binding of RFX3, another neurodevelopmental disorder risk gene, depends upon RFX4 to regulate synaptic gene expression. Furthermore, we identified lineage-specific functions for RFX4 in regulating proliferation during cortical inhibitory neuron development. Ultimately, we demonstrated that RFX4 deficiency persistently dysregulates neuronal gene expression through neuronal differentiation and disrupts cortical neuron stratification in organoid models. These consequences were absent in neurons generated by direct differentiation, confirming that neuronal phenotypes resulted from unconstrained neurogenesis. Finally, we modeled pathogenic missense mutation of the RFX4 DNA-binding domain. While this mutation strongly reduced DNA binding, it dysregulated synaptic gene expression distinctly from our deficiency models, supporting pathogenic mechanisms distinct from haploinsufficiency. Together, this work identified both shared and lineage-specific requirements for RFX4 during cortical development, building a necessary foundation for elucidating the etiology of RFX4-associated disorders.