Abstract: Over the past decade, the issue of AI governancehas emerged as a focal and high-profile concern in both national and global technology governance. Despite the repeated release of related governance consensus at various national and high-level global forums, implementation has been sluggish and effectiveness limited. The reason is that these efforts have focused on forming a centralized governance model based on laws and regulations with universal applicability. However, such a model fails to align with the intrinsic characteristics of AI development and the diverse intentions of societal stakeholders. In other words, there is a lack of incentive compatibility mechanisms. This paper introduces the concept of distributed AI governance:a decentralized AI compliance community predicated on technological mechanisms to ensure mutual trust and adherence to shared rules. Such a governance paradigm aligns more coherently with both the logic of AI technology and societal preferences. The core of the analysis lies in examining the incentive and constraint mechanisms that promote the broad inclusion of heterogeneous stakeholders into distributed governance. These mechanisms encompass alignment with societal values, maintenance of corporate reputation, market competition pressures, consensus within technical communities, and the assurance capabilities of technological infrastructure. In the future, AI governance should encourage groups with the same willingness and ability to form local governance communities with diversified scales and rules, and these communities should constitute a distributed governance network throughout the whole society, converging the will and actions of millions of subjects, and accumulating small goodness into big goodness. Simultaneously, the indispensable role of centralized governance anchored in public authority must be preserved to effectively govern AI behaviors that provoke serious safety, ethical, or value-based concerns.