diff --git a/packages/firestore/.gemini/commands/docs/amend.toml b/packages/firestore/.gemini/commands/docs/amend.toml new file mode 100644 index 00000000000..13562a6044f --- /dev/null +++ b/packages/firestore/.gemini/commands/docs/amend.toml @@ -0,0 +1,13 @@ +description = "Updates code documentation with new content" +prompt = """ + +Your primary role is to help explain this codebase and update the existing documentation with useful information for future maintainers. + +1. **Understand the existing documentation** Read everything in @devdocs/ to get a high level understanding of the concepts and core components of this codebase. Look for any relevant information on {{args.concept}}. +2. **Perform archeology on the codebase** Read everything in {{args.file_or_folder}} and learn more about {{args.concept}}. +3. **Verify your understanding of the concept** Stop and give a high level summary on {{args.concept}} is. Ask me questions to ensure you and and I are aligned on the code, the concept, why it exists, and the intended intended use. Create a high level summary of what you learned. Repeat asking me questions and summarizing until I tell you I'm ready to document what you know. +4. **Think carefully if and how to document this** Think deeply about what you know and if it is worth documenting. If so, think about the best way to document it. Documentation files, code comments, or both. Prefer high level documentation. +5. **Recommend a documentation plan** Re-read the documentation in @devdocs/ and recommend changes that would be helpful for future maintainers new to the codebase. Changes can include edits to the existing documentation or the creation of new documentation files. Edits to existing files must align to their purpose and level of detail. When creating new files, consider linking to it from existing documentation files for discovery. The content must be high level concepts with minimal references to code. Iterate with me until I'm satisfied with the proposed edits. + +Your final output is a recommendation to the documentation. +""" \ No newline at end of file diff --git a/packages/firestore/GEMINI.md b/packages/firestore/GEMINI.md new file mode 100644 index 00000000000..e4ce4c21237 --- /dev/null +++ b/packages/firestore/GEMINI.md @@ -0,0 +1,5 @@ +# Firestore JavaScript SDK +This project is the official JavaScript SDK for the [Google Cloud Firestore](https://firebase.google.com/docs/firestore) database. + +You are an expert in @devdocs/prerequisites.md +@devdocs/overview.md \ No newline at end of file diff --git a/packages/firestore/devdocs/architecture.md b/packages/firestore/devdocs/architecture.md new file mode 100644 index 00000000000..d839a217987 --- /dev/null +++ b/packages/firestore/devdocs/architecture.md @@ -0,0 +1,104 @@ +# SDK Architecture + +This document provides a detailed explanation of the Firestore JavaScript SDK's architecture, its core components, and the flow of data through the system. + +## Core Components + +The SDK is composed of several key components that work together to provide the full range of Firestore features. + +![Architecture Diagram](./architecture.png) + +* **API Layer**: The public-facing API surface that developers use to interact with the SDK. This layer is responsible for translating the public API calls into the internal data models and passing them to the appropriate core components. +* **Core**: + * **Event Manager**: Acts as a central hub for all eventing in the SDK. It is responsible for routing events between the API Layer and Sync Engine. It manages query listeners and is responsible for raising snapshot events, as well as handling connectivity changes and some query failures. + * **Sync Engine**: The central controller of the SDK. It acts as the glue between the Event Manager, Local Store, and Remote Store. Its responsibilities include: + * Coordinating client requests and remote events. + * Managing a view for each query, which represents the unified view between the local and remote data stores. + * Notifying the Remote Store when the Local Store has new mutations that need to be sent to the backend. +* **Local Store**: A container for the components that manage persisted and in-memory data. + * **Remote Table**: A cache of the most recent version of documents as known by the Firestore backend. + * **Mutation Queue**: A queue of all the user-initiated writes (set, update, delete) that have not yet been acknowledged by the Firestore backend. + * **Local View**: A cache that represents the user's current view of the data, combining the Remote Table with the Mutation Queue. + * **Overlays**: A performance-optimizing cache that stores the calculated effect of pending mutations from the Mutation Queue on documents. Instead of re-applying mutations every time a document is read, the SDK computes this "overlay" once and caches it, allowing the Local View to be constructed more efficiently. +* **Remote Store**: The component responsible for all network communication with the Firestore backend. It manages the gRPC streams for reading and writing data, and it abstracts away the complexities of the network protocol from the rest of the SDK. +* **Persistence Layer**: The underlying storage mechanism used by the Local Store to persist data on the client. In the browser, this is implemented using IndexedDB. + +The architecture and systems within the SDK map closely to the directory structure, which helps developers navigate the codebase. Here is a mapping of the core components to their corresponding directories. + +* `src/`: + * `api/`: Implements the **API Layer** for the main SDK. + * `lite-api/`: Implements the **API Layer** for the lite SDK. + * `core/`: Implements the **Sync Engine** and **Event Manager**. + * `local/`: Implements the **Local Store**, which includes the **Mutation Queue**, **Remote Table**, **Local View**, **Overlays** and the **Persistence Layer**. + * `remote/`: Implements the **Remote Store**, handling all network communication. + +For a more detailed explanation of the contents of each directory, see the [Code Layout](./code-layout.md) documentation. + +## Overview of features + +At a high level, all interactions with Firestore can be categorized as either reading or writing data. The SDK provides different mechanisms for these operations, each with distinct guarantees and performance characteristics. There is also a special case of writing data called tansactions detailed below. + + +### Read Operations + +There are two fundamental ways to read data from Firestore: + +* **One-Time Reads**: This is for fetching a snapshot of data at a specific moment. It's a simple request-response model. You ask for a document or the results of a query, and the server sends back the data as it exists at that instant. + +* **Real-Time Listeners**: This allows you to subscribe to a document or a query. The server first sends you the initial data and then continues to push updates to your client in real time as the data changes. This is the foundation of Firestore's real-time capabilities. + +When a query is executed, the SDK immediately returns data from the local cache, which includes any pending optimistic writes from the **Mutation Queue**. This provides a fast, responsive experience. At the same time, the SDK sends the query to the Firestore backend to fetch the latest version of the documents. When the fresh documents arrive from the backend, the SDK takes these server-authoritative documents and re-applies any pending mutations from the local queue on top of them. It then re-runs the original query against this newly merged data. If the documents still match the query's criteria, they are delivered to the query listener again. This is a common occurrence and means a listener could see an event for the same document twice: first with the cached, optimistic data, and a second time after the backend data is reconciled. + +### Write Operations + +All data modifications—creates, updates, and deletes—are treated as "writes." The SDK is designed to make writes atomic and resilient. There are two fundamental ways to write data to Firestore: + +* **One-Time Writes**: When a user performs a write (create, update, or delete), the operation is not sent directly to the backend. Instead, it's treated as a "mutation" and added to the local **Mutation Queue**. The SDK "optimistically" assumes the write will succeed on the backend and immediately reflects the change in the local view of the data, making the change visible to local queries. The SDK then works to synchronize this queue with the backend. This design is crucial for supporting offline functionality, as pending writes can be retried automatically when network connectivity is restored. + +* **Transactions**: For grouping multiple write operations into a single atomic unit, the SDK provides `runTransaction`. Unlike standard writes, transactions do not use the optimistic, offline-capable write pipeline. Instead, they are sent directly to the backend, which requires an active internet connection. This ensures atomicity but means transactions do not benefit from the offline capabilities of the standard write pipeline. + +### Data Bundles + +A Firestore data bundle is a serialized collection of documents and query results, created on a server using the Firebase Admin SDK. Bundles are used to efficiently deliver a pre-packaged set of data to the client, which can then be loaded directly into the SDK's local cache. This is useful for: + +* **Seeding initial data** for an application, allowing users to have a complete offline experience on their first use. +* **Distributing curated datasets** to clients in a single, efficient package. + +When a bundle is loaded, its contents are unpacked and stored in the local cache, making the data available for immediate querying without needing to connect to the Firestore backend. For more details, see the [Bundles documentation](./bundles.md). + + +# Data Flow + +Here's a step-by-step walkthrough of how data flows through the SDK for a write operation, referencing the core components. + +## Write Data Flow + +1. **API Layer**: A user initiates a write operation (e.g., `setDoc`, `updateDoc`, `deleteDoc`). +2. **Sync Engine**: The call is routed to the Sync Engine, which wraps the operation in a "mutation". +3. **Mutation Queue (in Local Store)**: The Sync Engine adds this mutation to the Mutation Queue. The queue is persisted to the **Persistence Layer** (IndexedDB). At this point, the SDK "optimistically" considers the write successful locally. +4. **Local View (in Local Store)**: The change is reflected in the Local View. This is done by creating or updating a cached **Overlay** for the affected document, making the change efficiently available to any active listeners without waiting for backend confirmation. +5. **Remote Store**: The Sync Engine notifies the Remote Store that there are pending mutations. +6. **Backend**: The Remote Store sends the mutations from the queue to the Firestore backend. +7. **Acknowledgement**: The backend acknowledges the write. +8. **Mutation Queue (in Local Store)**: The Remote Store informs the Sync Engine, which then removes the acknowledged mutation from the Mutation Queue. + +## Read Data Flow (with a Real-Time Listener) + +1. **API Layer**: A user attaches a listener to a query (e.g., `onSnapshot`). +2. **Event Manager**: The Event Manager creates a listener and passes it to the Sync Engine. +3. **Sync Engine**: The Sync Engine creates a "view" for the query. +4. **Local View (in Local Store)**: The Sync Engine asks the Local Store for the current documents matching the query. The Local Store provides these by applying cached **Overlays** on top of the documents to reflect optimistic local changes from the **Mutation Queue**. +5. **API Layer**: The initial data from the Local View is sent back to the user's `onSnapshot` callback. This provides a fast, initial result. +6. **Remote Store**: Simultaneously, the Sync Engine instructs the Remote Store to listen to the query on the Firestore backend. +7. **Backend**: The backend returns the initial matching documents for the query. +8. **Remote Table (in Local Store)**: The Remote Store receives the documents and saves them to the Remote Table in the Local Store, overwriting any previously cached versions of those documents. +9. **Sync Engine**: The Sync Engine is notified of the updated documents. It re-calculates the query view by combining the new data from the Remote Table with any applicable pending mutations from the **Mutation Queue**. +10. **API Layer**: If the query results have changed after this reconciliation, the new results are sent to the user's `onSnapshot` callback. This is why a listener may fire twice initially. +11. **Real-time Updates**: From now on, any changes on the backend that affect the query are pushed to the Remote Store, which updates the Remote Table, triggering the Sync Engine to re-calculate the view and notify the listener. + +## Bundle Loading Data Flow + +1. **API Layer**: The user initiates a bundle load via the public API. +2. **Sync Engine**: The Sync Engine receives the bundle and begins processing it. +3. **Local Store**: The Sync Engine unpacks the bundle and saves its contents (documents and named queries) into the **Local Store**. +4. **API Layer**: The user is notified of the progress and completion of the bundle loading operation via the task returned by the API. \ No newline at end of file diff --git a/packages/firestore/devdocs/architecture.png b/packages/firestore/devdocs/architecture.png new file mode 100644 index 00000000000..ec1fc04010b Binary files /dev/null and b/packages/firestore/devdocs/architecture.png differ diff --git a/packages/firestore/devdocs/build.md b/packages/firestore/devdocs/build.md new file mode 100644 index 00000000000..55b5e7b395f --- /dev/null +++ b/packages/firestore/devdocs/build.md @@ -0,0 +1,3 @@ +# Build Process + +This document provides a detailed explanation of the Firestore JavaScript SDK build process for the main and lite packages. diff --git a/packages/firestore/devdocs/bundles.md b/packages/firestore/devdocs/bundles.md new file mode 100644 index 00000000000..4e4256df324 --- /dev/null +++ b/packages/firestore/devdocs/bundles.md @@ -0,0 +1,42 @@ +# Firestore Data Bundles + +This document provides an overview of Firestore data bundles, how they are processed, and how they are used within the SDK. + +## What is a Bundle? + +A Firestore data bundle is a serialized, read-only collection of documents and named query results. Bundles are created on a server using the Firebase Admin SDK and can be efficiently distributed to clients. + +While bundles can be used for several purposes, their primary design motivation is to optimize Server-Side Rendering (SSR) workflows. In an SSR setup, a server pre-renders a page with data from Firestore. This data can be packaged into a bundle and sent to the client along with the HTML. The client-side SDK can then load this bundle and "hydrate" a real-time query with the pre-existing data, avoiding the need to re-fetch the same documents from the backend. This results in a significant performance improvement and cost savings. + +## Primary Use Case: Server-Side Rendering (SSR) Hydration + +The main workflow for bundles is as follows: + +1. **Server-Side:** A server fetches data from Firestore to render a page. +2. **Bundling:** The server packages the fetched documents and the corresponding query into a bundle. +3. **Transmission:** The bundle is embedded in the HTML page sent to the client. +4. **Client-Side:** The client-side JavaScript calls `loadBundle()` to load the data from the bundle into the SDK's local cache. +5. **Hydration:** The client then attaches a real-time listener to the same query that was bundled. The SDK finds the query results in the local cache and immediately fires the listener with the initial data, avoiding a costly roundtrip to the backend. + +## Other Benefits and Use Cases + +Beyond SSR hydration, bundles offer several other advantages: + +* **Enhanced Offline Experience:** Bundles can be shipped with an application's initial assets, allowing users to have a more complete offline experience from the first launch, reducing the need to sync every document individually. +* **Efficient Data Distribution:** They provide an efficient way to deliver curated or static datasets to clients in a single package. For instance, an application could bundle a list of popular items or configuration data. + +## The Loading Process + +When an application provides a bundle to the SDK, a loading process is initiated. The SDK reads the bundle, which is a stream of documents and queries, and saves each item into its local cache. This process is asynchronous, allowing the application to continue running while the data is being loaded in the background. + +To give developers visibility into this process, the SDK provides progress updates, including the number of documents and bytes loaded so far. Once all the data has been successfully loaded into the cache, the SDK signals that the process is complete. + +## Error Handling + +The bundle loading process is designed to be robust. If an error is encountered at any point—for example, if the bundle data is malformed or there is an issue writing to the local cache—the entire operation is aborted. The SDK ensures that the application is notified of the failure, allowing developers to catch the error and implement appropriate fallback or recovery logic. + +## Interacting with Bundled Data + +Once a bundle has been loaded, the data it contains is available for querying. If the bundle included named queries, you can use the `getNamedQuery()` method to retrieve a `Query` object, which can then be executed. + +When a named query is executed, the Firestore SDK will first attempt to fulfill the query from the local cache. If the results for the named query are available in the cache (because they were loaded from a bundle), they will be returned immediately, without a server roundtrip. diff --git a/packages/firestore/devdocs/code-layout.md b/packages/firestore/devdocs/code-layout.md new file mode 100644 index 00000000000..0bdd6e4a38f --- /dev/null +++ b/packages/firestore/devdocs/code-layout.md @@ -0,0 +1,23 @@ +# SDK Code Layout + +This document explains the code layout in this repository. It is closely related to the [architecture](./architecture.md). + +* `src/`: Contains the source code for the main `@firebase/firestore` package. + * `api/`: Implements the **API Layer** for the main SDK. + * `lite-api/`: Contains the entry point of for the lite SDK. + * `core/`: Contains logic for the **Sync Engine** and **Event Manager**. + * `local/`: Contains the logic the **Local Store**, which includes the **Mutation Queue**, **Remote Table**, **Local View**, **Overlays**, and the **Persistence Layer**. + * `remote/`: Contains the logic for the **Remote Store**, handling all network communication. + * `model/`: Defines the internal data models used throughout the SDK, such as `Document`, `DocumentKey`, and `Mutation`. These models are used to represent Firestore data and operations in a structured way. + * `platform/`: Contains platform-specific code to abstract away the differences between the Node.js and browser environments. This includes things like networking, storage, and timers. This allows the core logic of the SDK to be platform-agnostic. + * `protos/`: Contains the Protocol Buffer (`.proto`) definitions that describe the gRPC API surface of the Firestore backend. These files are used to generate the client-side networking code. +* `lite/`: Defines the entrypoint code for the `@firebase/firestore/lite` package. +* `test/`: Contains all unit and integration tests for the SDK. The tests are organized by component and feature, and they are essential for ensuring the quality and correctness of the code. +* `scripts/`: Contains a collection of build and maintenance scripts used for tasks such as bundling the code, running tests, and generating documentation. + +TODO: Add more detailed information as appropriate on each folder + +TODO: Mention critical entry points + - `package.json` for packages and common commands. Go to [build.md](./build.md) for details + - rollup configs for main and lite sdks. Go to [build.md](./build.md) for details + - tests entry points. Go to [testing.md](./testing.md) for details \ No newline at end of file diff --git a/packages/firestore/devdocs/overview.md b/packages/firestore/devdocs/overview.md new file mode 100644 index 00000000000..6f4f52a437c --- /dev/null +++ b/packages/firestore/devdocs/overview.md @@ -0,0 +1,45 @@ +# Firestore JavaScript SDK Overview + +This document is the starting point for navigating the Firestore JavaScript SDK codebase and its documentation. It provides a high-level overview of the SDK, its architecture, how it is built, tested, and the developer workflow. + +All contributors are expected to be familiar with the [prerequisites](./prerequisites.md) before working in this codebase. + +## Project Goals + +The Firestore JavaScript SDK is one of the official client-side library for interacting with [Google Cloud Firestore](https://firebase.google.com/docs/firestore). It is designed to be used in a variety of JavaScript environments, including web browsers (primary and common) and Node.js (secondary and rare). It is important to distinguish this SDK from the [Google Cloud Firestore server-side SDK for Node.js](https://github.com/googleapis/nodejs-firestore). While this SDK can run in Node.js, it is primarily designed for client-side use. The server-side SDK is intended for trusted environments and offers different capabilities. However, the two SDKs are designed to harmonize where helpful (e.g. data models) to facilitate easier full-stack application development. + +The primary goals of this SDK are: + +* Provide a simple and intuitive API for reading and writing data to Firestore. +* Support real-time data synchronization with streaming queries. +* Enable offline data access and query caching. +* Offer a lightweight version for applications that do not require advanced features. +* Maintain API and architectural symmetry with the [Firestore Android SDK](https://github.com/firebase/firebase-android-sdk) and [Firestore iOS SDK](https://github.com/firebase/firebase-ios-sdk). This consistency simplifies maintenance and makes it easier to port features between platforms. The public API is intentionally consistent across platforms, even if it means being less idiomatic, to allow developers to more easily port their application code. + +## Artifacts + +The Firestore JavaScript SDK is divided into two main packages: + +* `@firebase/firestore`: The main, full-featured SDK that provides streaming and offline support. +* `@firebase/firestore/lite`: A much lighter-weight (AKA "lite") version of the SDK for applications that do not require streaming or offline support. + +For a detailed explanation of the architecture, components, and data flow, please see the [Architecture documentation](./architecture.md). Related, for a deailed overview of the source code layout, please see [Code layout](./code-layout.md). + + +## Build + +TODO: Add critical information about the build process including optimizations for code size, etc. + +For information on how the artifacts are built, please see the [Build documentation](./build.md) file. + +## Testing + +TODO: Add critical information about the tests harness, organization, spec tests, etc. + +For information on how the tests are setup and organized [Testing documentation](./testing.md) file. + +## Developer Workflow + +TODO: Add list of common commands here. + +For information on the developer workflow, including how to build, test, and format the code, please see the [CONTRIBUTING.md](../CONTRIBUTING.md) file. diff --git a/packages/firestore/devdocs/prerequisites.md b/packages/firestore/devdocs/prerequisites.md new file mode 100644 index 00000000000..dabe19d7b90 --- /dev/null +++ b/packages/firestore/devdocs/prerequisites.md @@ -0,0 +1,31 @@ +# Firestore JavaScript SDK Maintainer's Guide + +This document outlines the prerequisite knowledge for new maintainers of the Firestore JavaScript SDK. + +## Prerequisite Knowledge + +Before contributing to this codebase, you should have a strong understanding of the following technologies and concepts: + +### Core Technologies + +* **TypeScript:** The entire codebase is written in TypeScript. A deep understanding of TypeScript, including its type system, generics, and modules, is essential. +* **JavaScript (ES6+):** As a JavaScript SDK, a strong grasp of modern JavaScript features is required. +* **Node.js:** The SDK is isomorphic and runs in the Node.js environment. Familiarity with Node.js concepts, such as its module system and event loop, is important. +* **Browser Runtime Environment:** The SDK is also used in web browsers. A good understanding of the different browser execution contexts (e.g. main window, web/service workers) and subsystems (e.g. persistence like IndexedDB and Local Storage, networking) is necessary. + +### Build and Test Tooling + +* **Yarn:** We use Yarn for package management. You should be familiar with basic Yarn commands. +* **Rollup.js:** Our build process uses Rollup.js to bundle the code. Understanding Rollup's configuration and plugin system will be helpful. +* **Karma, Mocha, and Chai:** These are our testing frameworks. You should be comfortable writing and running tests using this stack. + + + +### Domain Knowledge + +* **[Google Cloud Firestore](https://firebase.google.com/docs/firestore):** A deep understanding of Firestore's data model (documents, collections, subcollections), query language, and security rules is fundamental. +* **Databases:** A general understanding of databases, including key-value stores and relational databases, is helpful for understanding Firestore's design and trade-offs. +* **Modern Web Application Architecture:** Familiarity with modern web application architecture and also server-side rendering (SSR), is beneficial for understanding how the SDK is used in practice. +* **[Firebase](https://firebase.google.com/docs):** Familiarity with the Firebase platform is required, especially Firebase Auth and Firebase Functions. +* **Protocol Buffers / gRPC:** The main SDK uses Protocol Buffers over gRPC to communicate with the Firestore backend. A basic understanding of these technologies is helpful. +* **Firestore REST API:** The lite SDK uses the Firestore REST API. Familiarity with the REST API is useful when working on the lite version of the SDK. diff --git a/packages/firestore/devdocs/testing.md b/packages/firestore/devdocs/testing.md new file mode 100644 index 00000000000..11f00ebca84 --- /dev/null +++ b/packages/firestore/devdocs/testing.md @@ -0,0 +1,15 @@ +# Build Process + +This document provides a detailed explanation of the Firestore JavaScript SDK testing strategy, tech stack, and patterns and practices. + +# Tech Stack +- karma, mocha, chai + +# Strategy +- Firebase emulator for local development +- Integration testing with the backend + +# Patterns and Practices + + +# Spec Tests \ No newline at end of file