Documentation

Tokens

Search tokens in dataset

Search through tokens with mixed token/tag sequence matching.

ProductionStagingUATLocal development
GEThttps://linguistic.ankiren.com/api/datasets/{slug}/tokens

Parameters

Path Parameters

  • slugstringpathRequired

    URL-friendly identifier for the dataset

Query Parameters

  • searchstringquery

    Search conditions as comma-separated values. Use tag:X prefix for POS tag matching, plain text for token matching.

  • split"train" | "test"query

    Filter by data split (train or test)

  • pageintegerquery

    Page number (1-indexed)

    default: 1min: 1
  • limitintegerquery

    Results per page (max 100)

    default: 50min: 1max: 100

Responses

200 · Token search results with sentence context

  • tokens[TokenWithSentence]Required
    • sentenceIdintegerRequired

      Sentence index within the dataset

    • positionintegerRequired

      Token position within the sentence (0-indexed)

    • tokenstringRequired

      The matched token text

    • tagstringRequired

      POS tag of the token

    • split"train" | "test"Required

      Data split (train or test)

    • sentencestringRequired

      Full sentence text (space-joined tokens)

    • sentenceTokens[object]Required

      All tokens in the sentence with their tags

    • matchPositions[integer]

      Positions of all matched tokens in the sequence

    • matches[object]

      Detailed match information with types

  • totalintegerRequired

    Total number of matching sentences

  • pageintegerRequired

    Current page number

  • limitintegerRequired

    Results per page

  • totalPagesintegerRequired

    Total number of pages

  • datasetobjectRequired

    Dataset metadata summary

    • idinteger
    • slugstring
    • namestring
application/json
{
  "tokens": [
    {
      "sentenceId": 42,
      "position": 3,
      "token": "đi",
      "tag": "V",
      "split": "train",
      "sentence": "Tôi muốn đi ra ngoài",
      "sentenceTokens": [
        {
          "token": "Tôi",
          "tag": "P"
        },
        {
          "token": "muốn",
          "tag": "V"
        },
        {
          "token": "đi",
          "tag": "V"
        },
        {
          "token": "ra",
          "tag": "R"
        },
        {
          "token": "ngoài",
          "tag": "N"
        }
      ],
      "matchPositions": [
        2,
        3
      ],
      "matches": [
        {
          "position": 2,
          "type": "token"
        },
        {
          "position": 3,
          "type": "tag"
        }
      ]
    }
  ],
  "total": 1523,
  "page": 1,
  "limit": 50,
  "totalPages": 31,
  "dataset": {
    "id": 1,
    "slug": "vlsp2013-postag",
    "name": "VLSP2013_POSTAG"
  }
}

Sample Code

Đổi ngôn ngữ ở ô Language trên thanh Documentation.

cURL
curl -X GET 'https://linguistic.ankiren.com/api/datasets/vlsp2013-postag/tokens' \
  -H 'Authorization: Bearer <token>' \
  -H 'Accept: application/json'