find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/semantic-scholar:semantic-scholar-incremental-updates-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.2.0
info:
  title: S2AG Datasets Incremental Updates API
  version: '1.0'
  description: 'Download full-corpus datasets from the Semantic Scholar Academic Graph (S2AG)


    Some python demonstrating usage of the datasets API:


    r1 = requests.get(''https://api.semanticscholar.org/datasets/v1/release'').json()

    print(r1[-3:])

    [''2023-03-14'', ''2023-03-21'', ''2023-03-28'']


    r2 = requests.get(''https://api.semanticscholar.org/datasets/v1/release/latest'').json()

    print(r2[''release_id''])

    2023-03-28


    print(json.dumps(r2[''datasets''][0], indent=2))

    {

    "name": "abstracts",

    "description": "Paper abstract…'
servers:
- url: https://api.semanticscholar.org/datasets/v1
tags:
- name: Incremental Updates
paths:
  /diffs/{start_release_id}/to/{end_release_id}/{dataset_name}:
    get:
      summary: Download Links for Incremental Diffs
      operationId: get_diff
      tags:
      - Incremental Updates
      description: 'Full datasets can be updated from one release to another to avoid

        downloading and processing data that hasn''t changed. This method returns

        a list of all the "diffs" that are required to catch a given dataset up

        from its current release to a newer one.


        Each "diff" represents changes between two sequential releases, and

        contains two lists of files, an "updated" list and a "deleted" list.

        Records in the "updated" list need to be inserted or replaced by their

        primary key. Records in the "deleted" list should be removed.


        Example code for updating a database or key/value store:


        difflist = requests.get(''https://api.semanticscholar.org/datasets/v1/diffs/2023-08-01/to/latest/papers'').json()

        for diff in difflist[''diffs'']:

        for url in diff[''update_files'']:

        for json_line in requests.get(url).iter_lines():

        record = json.loads(json_line)

        datastore.upsert(record[''corpusid''], record)

        for url in diff[''delete_files'']:

        for json_line in requests.get(url).iter_lines():

        record = json.loads(json_line)

        datastore.delete(record[''corpusid''])


        Example code for updating via a join in Spark:


        current = sc.textFile(''s3://curr-dataset-location'').map(json.loads).keyBy(lambda x: x[''corpusid''])

        updates = sc.textFile(''s3://diff-updates-location'').map(json.loads).keyBy(lambda x: x[''corpusid''])

        deletes = sc.textFile(''s3://diff-deletes-location'').map(json.loads).keyBy(lambda x: x[''corpusid''])


        updated = current.fullOuterJoin(updates).mapValues(lambda x: x[1] if x[1] is not None else x[0])

        updated = updated.fullOuterJoin(deletes).mapValues(lambda x: None if x[1] is not None else x[0]).filter(lambda x: x[1] is not None)

        updated.values().map(json.dumps).saveAsTextFile(''s3://updated-dataset-location'')'
      responses:
        '200':
          description: List of download links for one dataset between given releases
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Dataset%20Diff%20List'