feat: data pipeline changes for segmentRepositories [IN-552] [IN-554]

[borfast]

In the context of IN-524, we want to have the archived and excluded repositories for each project returned along with the project data.

This means adding the repository data to TinyBird, which already includes the archived and excluded flags in the segmentRepositories table, and then merge that data into the existing pipes that return project data, namely the projects_list and the search_collections_projects_repos pipes.

This PR:

• Adds the database migration to ensure we get data from the segmentRepositories table into Sequin.
• Adds a new segmentRepositories data source in TinyBird to receive data from the aforementioned table.
• Modifies the insights_projects_populated_copy pipe to include the list of archived and excluded repositories, which is then propagated to the insights_projects_populated_ds data source, from there it feeds the insightsProjects_filtered pipe, and finally the projects_list pipe, which is the end goal.
  • The search_collections_projects_repos contains archived and excluded columns for the repositories results. This required adding a JOIN to the query, and with all the optimisations that were done not too long ago, I'm not sure if this

Specific tickets in Jira:

• https://linuxfoundation.atlassian.net/browse/IN-552
• https://linuxfoundation.atlassian.net/browse/IN-554

[github-actions]

Conventional Commits FTW!

[Copilot]

Pull Request Overview

This PR implements the infrastructure to propagate archived and excluded repository data from the segmentRepositories table through the TinyBird pipeline to support enhanced project filtering and search capabilities.

• Adds database migration to enable replication of the segmentRepositories table to TinyBird
• Creates a new segmentRepositories data source in TinyBird and updates pipeline to aggregate archived/excluded repository lists
• Propagates archived repositories data through the insights project pipeline to enable repository-aware search functionality

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
backend/src/database/migrations/V1753798345__segmentRepositories_replication.sql	Enables replication of segmentRepositories table to TinyBird
services/libs/tinybird/datasources/segmentRepositories.datasource	Defines TinyBird data source schema for segmentRepositories table
services/libs/tinybird/datasources/insights_projects_populated_ds.datasource	Adds archived/excluded repositories fields to project data source
services/libs/tinybird/pipes/insights_projects_populated_copy.pipe	Aggregates archived/excluded repositories and joins to project data
services/libs/tinybird/pipes/insightsProjects_filtered.pipe	Passes through archived/excluded repositories to filtered results
services/libs/tinybird/pipes/search_collections_projects_repos.pipe	Exposes archived repositories in search results

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[epipav]

we don't need this grant - sequin user only works on the sequin schema

[borfast]

Doesn't it need access to do the backfills? I confess I never checked how it does them, but I assumed it was just querying the database normally.

[epipav]

Instead of using Nullable fields, it's better to use Non-nullable fields with default values for performance reasons

[borfast]

The column is a datetime and nullable on Postgres, and I think we might have had similar situations before - how are we converting null rows to non-null for Tinybird? Do we do a transformation in Sequin, for example?

[epipav]

You'll need an updatedAt field to keep track of different versions of the same row. createdAt won't change on consecutive updates to the same row (when it's archived or excluded, for instance). So it can even pick the old row when an update happens

[borfast]

Oh, yeah, I forgot about that. The original table Umberto created doesn't have it and I wondered if it was necessary, but forgot to check it.

[borfast]

I added the column and also updated the worker code to update its timestamp whenever it updates a repository archived / excluded status.