Add --issue-timeline flag and refresh timelines when cross-references change (#168)

README.rst

@@ -41,7 +41,8 @@ CLI Help output::
                   [--incremental-by-files]
                   [--starred] [--all-starred] [--starred-skip-size-over MB]
                   [--watched] [--followers] [--following] [--all]
-                  [--issues] [--issue-comments] [--issue-events] [--pulls]
+                  [--issues] [--issue-comments] [--issue-events]
+                  [--issue-timeline] [--pulls]
                   [--pull-comments] [--pull-reviews] [--pull-commits]
                   [--pull-details]
                   [--labels] [--hooks] [--milestones] [--security-advisories]
@@ -96,6 +97,8 @@ CLI Help output::
       --issues              include issues in backup
       --issue-comments      include issue comments in backup
       --issue-events        include issue events in backup
+      --issue-timeline      include issue timeline in backup (cross-references,
+                            commits and reviews; complements --issue-events)
       --pulls               include pull requests in backup
       --pull-comments       include pull request review comments in backup
       --pull-reviews        include pull request reviews in backup
@@ -347,6 +350,27 @@ For finer control, avoid using ``--assets`` with starred repos, or use ``--skip-
 
 Alternatively, consider just storing links to starred repos in JSON format with ``--starred``.
 
+
+About issue events and the issue timeline
+-----------------------------------------
+
+Use ``--issue-events`` and ``--issue-timeline`` with ``--issues``; on their own they back up nothing.
+
+``--issue-events`` backs up GitHub's issue events endpoint under each issue's ``event_data`` key. That endpoint returns a restricted set of event types and **never** includes cross-references, so an issue that was mentioned from another issue or pull request shows no sign of it. GitHub only exposes those through the timeline.
+
+``--issue-timeline`` backs up the timeline endpoint under a separate ``timeline_data`` key. The two flags are independent: ``event_data`` keeps its existing contents and shape whether or not the timeline is enabled, so nothing that already reads it is affected.
+
+Use both flags together for a complete picture. Neither endpoint is a superset of the other, so they are complementary rather than alternatives. Only the timeline reports ``cross-referenced``, ``committed`` and ``reviewed`` entries. Only the events endpoint reliably reports ``referenced`` entries, where a commit elsewhere in the repository mentions the issue; the timeline omits many of these, substituting the issue's own ``committed`` entries instead.
+
+Note that timeline entries are not all the same shape as events. ``cross-referenced`` and ``committed`` entries have no ``id`` field, so anything consuming ``timeline_data`` should not assume one is present. ``commented`` entries are excluded from the backup because they embed the full comment body, which ``--issue-comments`` already saves.
+
+``--issue-timeline`` is included in ``--all``. On issues with many cross-references it can be noticeably larger and slower than ``--issue-events`` alone, because each ``cross-referenced`` entry embeds the referencing issue in full. On a heavily cross-referenced issue this can mean several times the API requests and many times the JSON on disk. Ordinary repositories see little difference.
+
+Incremental backups need extra help here, because being referenced from elsewhere does not change an issue's ``updated_at``. An issue can be mentioned from another repository years after its last activity, and the usual ``since`` listing will never report it as changed. On an incremental run with ``--issue-timeline``, github-backup therefore asks GitHub's GraphQL API for each item's cross-reference count and newest timestamp, compares those against what is already saved, and re-fetches the timeline of anything that differs. Pull requests are included in that check when they are being stored as issues, which is the case unless ``--pulls`` is used. The check needs no extra state on disk and also notices references that were later deleted. It costs one rate-limit point per 100 items checked, against GitHub's GraphQL quota, which is separate from the REST one. That is cheap in quota terms even on very large repositories, but it is not instant: a repository with 18,000 issues and pull requests took around 200 points and five minutes for the check alone.
+
+If you enable ``--issue-timeline`` on an existing incremental backup, cross-references on older issues are picked up automatically by that check. Timeline entries of other kinds, such as commits and reviews, are only fetched when the issue itself is updated, so run once without ``--incremental`` if you want those filled in for historical issues too.
+
+
 About pull request reviews
 --------------------------
 
@@ -450,14 +474,14 @@ Quietly and incrementally backup useful Github user data (public and private rep
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
 
 Debug an error/block or incomplete backup into a temporary directory. Omit "incremental" to fill a previous incomplete backup. ::
 
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
 
 Pipe a token from stdin to avoid storing it in environment variables or command history (Unix-like systems only)::
 

github_backup/github_backup.py

@@ -18,6 +18,7 @@
 import sys
 import threading
 import time
+import traceback
 from collections.abc import Generator
 from datetime import datetime
 from http.client import IncompleteRead
@@ -33,10 +34,14 @@
     VERSION = "unknown"
 
 from .graphql_queries import (
+    CROSS_REFERENCE_NODE_LIMIT,
+    CROSS_REFERENCE_PAGE_SIZE,
     DISCUSSION_DETAIL_QUERY,
     DISCUSSION_LIST_QUERY,
     DISCUSSION_PAGE_SIZE,
     DISCUSSION_REPLIES_QUERY,
+    ISSUE_CROSS_REFERENCE_COUNT_QUERY,
+    PULL_CROSS_REFERENCE_COUNT_QUERY,
 )
 
 FILE_URI_PREFIX = "file://"
@@ -278,6 +283,12 @@ def parse_args(args=None):
         dest="include_issue_events",
         help="include issue events in backup",
     )
+    parser.add_argument(
+        "--issue-timeline",
+        action="store_true",
+        dest="include_issue_timeline",
+        help="include issue timeline in backup (cross-references, commits and reviews; complements --issue-events)",
+    )
     parser.add_argument(
         "--pulls",
         action="store_true",
@@ -2569,6 +2580,141 @@ def backup_discussions(args, repo_cwd, repository):
         )
 
 
+def cross_reference_state(timeline_data):
+    """Return (count, newest timestamp) of the cross-references in a timeline."""
+    created = [
+        item.get("created_at")
+        for item in timeline_data or []
+        if item.get("event") == "cross-referenced"
+    ]
+    return len(created), max((c for c in created if c), default=None)
+
+
+def _cross_reference_page(args, repository, query, connection, use_total_count):
+    """Page one GraphQL connection, yielding {number: (count, newest)}.
+
+    totalCount honours the itemTypes filter on issues but not on pull
+    requests, so pull request cross-references are counted from the nodes.
+    """
+    owner, name = _repository_owner_name(repository)
+    state = {}
+    after = None
+    page = 1
+
+    while True:
+        variables = {
+            "owner": owner,
+            "name": name,
+            "after": after,
+            "pageSize": CROSS_REFERENCE_PAGE_SIZE,
+        }
+        if not use_total_count:
+            # Only the pull request query declares this.
+            variables["nodeLimit"] = CROSS_REFERENCE_NODE_LIMIT
+
+        data = retrieve_graphql_data(
+            args,
+            query,
+            variables,
+            log_context="cross-references {0} {1} page {2}".format(
+                repository["full_name"], connection, page
+            ),
+        )
+        repository_data = data.get("repository")
+        if repository_data is None:
+            raise Exception(
+                "Repository {0} not found in GraphQL response".format(
+                    repository["full_name"]
+                )
+            )
+
+        items = repository_data.get(connection) or {}
+        for node in _connection_nodes(items):
+            timeline = node.get("timelineItems") or {}
+            created = [n.get("createdAt") for n in timeline.get("nodes") or [] if n]
+            if use_total_count:
+                count = timeline.get("totalCount", 0)
+            elif len(created) >= CROSS_REFERENCE_NODE_LIMIT:
+                # Counted from the nodes, so it may be truncated. Compare on
+                # the timestamp alone rather than on a count we cannot trust.
+                count = None
+            else:
+                count = len(created)
+            state[node["number"]] = (count, created[-1] if created else None)
+
+        page_info = items.get("pageInfo") or {}
+        if not page_info.get("hasNextPage"):
+            break
+
+        after = page_info.get("endCursor")
+        page += 1
+
+    return state
+
+
+def retrieve_cross_reference_state(args, repository, include_pulls=False):
+    """Ask GitHub for each item's cross-reference count and newest timestamp."""
+    state = _cross_reference_page(
+        args,
+        repository,
+        ISSUE_CROSS_REFERENCE_COUNT_QUERY,
+        "issues",
+        use_total_count=True,
+    )
+    if include_pulls:
+        state.update(
+            _cross_reference_page(
+                args,
+                repository,
+                PULL_CROSS_REFERENCE_COUNT_QUERY,
+                "pullRequests",
+                use_total_count=False,
+            )
+        )
+    return state
+
+
+def find_stale_timeline_issues(args, issue_cwd, repository, include_pulls=False):
+    """Issue numbers whose stored cross-references no longer match GitHub's.
+
+    Incremental runs select issues by updated_at, which a cross-reference does
+    not change, so those issues would otherwise never be revisited.
+    """
+    if not get_graphql_auth(args):
+        # The check needs the GraphQL API, which always requires a token.
+        logger.debug("Skipping cross-reference check, no authentication")
+        return set()
+
+    try:
+        live = retrieve_cross_reference_state(args, repository, include_pulls)
+    except Exception as e:
+        logger.warning(
+            "Unable to check {0} for new cross-references, "
+            "issues referenced since the last backup may be missed: {1}".format(
+                repository["full_name"], e
+            )
+        )
+        logger.debug(traceback.format_exc())
+        return set()
+
+    stale = set()
+    for number, (count, newest) in live.items():
+        existing = read_json_file_if_exists("{0}/{1}.json".format(issue_cwd, number))
+        if existing is None:
+            continue
+        stored = cross_reference_state(existing.get("timeline_data"))
+        changed = stored[1] != newest if count is None else stored != (count, newest)
+        if changed:
+            stale.add(number)
+
+    if stale:
+        logger.info(
+            "Refreshing {0} issues with new cross-references".format(len(stale))
+        )
+        logger.debug("Stale timelines: {0}".format(sorted(stale)))
+    return stale
+
+
 def backup_issues(args, repo_cwd, repository, repos_template):
     has_issues_dir = os.path.isdir("{0}/issues/.git".format(repo_cwd))
     if args.skip_existing and has_issues_dir:
@@ -2584,6 +2730,7 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     _issue_template = "{0}/{1}/issues".format(repos_template, repository["full_name"])
 
     should_include_pulls = args.include_pulls or args.include_everything
+    include_timeline = args.include_issue_timeline or args.include_everything
     issue_states = ["open", "closed"]
     for issue_state in issue_states:
         query_args = {"filter": "all", "state": issue_state}
@@ -2600,6 +2747,30 @@ def backup_issues(args, repo_cwd, repository, repos_template):
 
             issues[issue["number"]] = issue
 
+    stale_timelines = set()
+    incremental_run = args.since or args.incremental_by_files
+    if (
+        include_timeline
+        and incremental_run
+        and any(f.endswith(".json") for f in os.listdir(issue_cwd))
+    ):
+        stale_timelines = find_stale_timeline_issues(
+            args, issue_cwd, repository, include_pulls=not should_include_pulls
+        )
+        for number in sorted(stale_timelines - set(issues)):
+            try:
+                issues[number] = retrieve_data(
+                    args, "{0}/{1}".format(_issue_template, number), paginated=False
+                )[0]
+            except (HTTPError, RepositoryUnavailableError, IndexError) as e:
+                # Deleted or transferred between the sweep and this fetch. The
+                # refresh is opportunistic, so skip rather than fail the run.
+                logger.warning(
+                    "Unable to refresh cross-references for issue {0}: {1}".format(
+                        number, e
+                    )
+                )
+
     if issues_skipped:
         issues_skipped_message = " (skipped {0} pull requests)".format(issues_skipped)
 
@@ -2610,9 +2781,14 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     )
     comments_template = _issue_template + "/{0}/comments"
     events_template = _issue_template + "/{0}/events"
+    timeline_template = _issue_template + "/{0}/timeline"
     for number, issue in list(issues.items()):
         issue_file = "{0}/{1}.json".format(issue_cwd, number)
-        if args.incremental_by_files and os.path.isfile(issue_file):
+        if (
+            args.incremental_by_files
+            and os.path.isfile(issue_file)
+            and number not in stale_timelines
+        ):
             modified = os.path.getmtime(issue_file)
             modified = datetime.fromtimestamp(modified).strftime("%Y-%m-%dT%H:%M:%SZ")
             if modified > issue["updated_at"]:
@@ -2629,6 +2805,16 @@ def backup_issues(args, repo_cwd, repository, repos_template):
         if args.include_issue_events or args.include_everything:
             template = events_template.format(number)
             issues[number]["event_data"] = retrieve_data(args, template)
+        if include_timeline:
+            template = timeline_template.format(number)
+            # "commented" timeline entries embed the full comment body, which
+            # --issue-comments already backs up; storing them twice invites the
+            # two copies to diverge.
+            issues[number]["timeline_data"] = [
+                item
+                for item in retrieve_data(args, template)
+                if item.get("event") != "commented"
+            ]
         if args.include_attachments:
             download_attachments(
                 args, issue_cwd, issues[number], number, repository, item_type="issue"

github_backup/graphql_queries.py

@@ -2,6 +2,66 @@
 
 DISCUSSION_PAGE_SIZE = 100
 
+CROSS_REFERENCE_PAGE_SIZE = 100
+
+# Cross-references do not change an issue's updatedAt, so the REST listing
+# cannot report them as changed. Count and newest timestamp are compared
+# against what is on disk: the count alone would miss an add paired with a
+# delete, the timestamp alone would miss a delete.
+ISSUE_CROSS_REFERENCE_COUNT_QUERY = """
+query($owner: String!, $name: String!, $after: String, $pageSize: Int!) {
+  repository(owner: $owner, name: $name) {
+    issues(first: $pageSize, after: $after) {
+      nodes {
+        number
+        timelineItems(last: 1, itemTypes: [CROSS_REFERENCED_EVENT]) {
+          totalCount
+          nodes {
+            ... on CrossReferencedEvent {
+              createdAt
+            }
+          }
+        }
+      }
+      pageInfo {
+        hasNextPage
+        endCursor
+      }
+    }
+  }
+}
+"""
+
+# Pull requests are stored as issues when --pulls is not used, and the issues
+# connection above excludes them. This query is shaped differently because
+# totalCount ignores itemTypes on a pull request, unlike on an issue, so the
+# cross-references have to be counted from the nodes themselves.
+CROSS_REFERENCE_NODE_LIMIT = 100
+
+PULL_CROSS_REFERENCE_COUNT_QUERY = """
+query($owner: String!, $name: String!, $after: String, $pageSize: Int!,
+      $nodeLimit: Int!) {
+  repository(owner: $owner, name: $name) {
+    pullRequests(first: $pageSize, after: $after) {
+      nodes {
+        number
+        timelineItems(last: $nodeLimit, itemTypes: [CROSS_REFERENCED_EVENT]) {
+          nodes {
+            ... on CrossReferencedEvent {
+              createdAt
+            }
+          }
+        }
+      }
+      pageInfo {
+        hasNextPage
+        endCursor
+      }
+    }
+  }
+}
+"""
+
 DISCUSSION_LIST_QUERY = """
 query($owner: String!, $name: String!, $after: String, $pageSize: Int!) {
   repository(owner: $owner, name: $name) {