Dev and Test

This document describes the organization of the source code for Coordinator and provides instructions on how to build Coordinator from source and run tests.

Dev Environment

To build Coordinator from source code, you need to prepare a development environment with many dependencies and build toolchains. You using the following ways to prepare the development environment:

  • Use our provided dev container

  • Install all tools and dependencies on your local machine.

We strongly recommend you to use the dev containers to develop and test.

Develop with dev containers

We provided a docker image registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-dev:latest-${arch} with all tools and dependencies included.

# x86_64
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-dev:latest-amd64
# inside the container
cd GraphScope && make coordinator

See doc for finding more detailed information on how to develop Coordinator from vscode.

Install deps on local

To install all dependencies on your local, use the GraphScope command-line utility gsctl with the subcommand install-deps like this

pip3 install gsctl
# install coordinator dependencies
# dev-learning is the minimal set of dependencies to build coordinator
gsctl install-deps dev-learning
# build
cd GraphScope && make coordinator

Understanding the Codebase

The Coordinator code is organized in the coordinator/gscoordinator/flex folder as follows:

  • controllers: Controllers that maps incoming HTTP requests to corresponding API endpoints defined in the OpenAPI specification.

  • core: Contains the core implementation coordinator

    • interactive: Core implementation of interaction with Flex Interactive

    • insight: Core implementation of interaction with Flex GraphScope Insight

  • models: Data Structures generated by OpenApi

  • openapi: OpenApi specification file

Testing with Interactive

Preparation

Typically, as the middle layer for interaction between client and engine, we need to deploy and test the Coordinator with a specific instance under the FLEX architecture, such as Interactive. Please make sure the engine service is deployed in advance according to the corresponding development documentation.

Launching Coordinator

# generate configuration file
cat > /tmp/config.yaml << EOF
coordinator:
  http_port: 8080

launcher_type: hosts

session:
  instance_id: demo
EOF

# start
export SOLUTION=INTERACTIVE
python3 -m gscoordinator --config-file /tmp/config.yaml

Build gsctl

This will install the gsctl package, thus make gsctl work.

Tip

This package would also be installed in editable mode, which means any changed you made in local directory will take effect.

make gsctl

Run Test Script

Numerous test cases have been created for Coordinator with Interactive, which can be referenced in the GitHub workflow flex-interactive.yaml.

# pip3 install pytest
# By default, it will connect to the coordinator service at 8080 port
cd GraphScope/python
python3 -m pytest -s -v ./graphscope/gsctl/tests/test_interactive.py

Running tests manually via gsctl

In most cases, you may want to test a specific API interface, such as data loading, or creating a storedProcedure. At this point you can use gsctl to connect to the coordinator.

gsctl connect --coordinator-endpoint http://127.0.0.1:8080
# change the port number if you have customized the coordinator port.

After connecting to the Coordinator Service, you can now view current service status. Always remember to --help on a command to get more information.

gsctl ls -l

See Restful API Doc for more detailed data structure information. For all test commands and parameter explanation, please refer to Custom Graph Data, here we just list some basic commands.

Create a New Graph

$ gsctl create graph -f ./modern_graph.yaml

$ cat modern_graph.yaml
name: test_graph
description: "This is a test graph"
schema:
  vertex_types:
    - type_name: person
      properties:
        - property_name: id
          property_type:
            primitive_type: DT_SIGNED_INT64
        - property_name: name
          property_type:
            string:
              long_text: ""
        - property_name: age
          property_type:
            primitive_type: DT_SIGNED_INT32
      primary_keys:
        - id
  edge_types:
    - type_name: knows
      vertex_type_pair_relations:
        - source_vertex: person
          destination_vertex: person
          relation: MANY_TO_MANY
      properties:
        - property_name: weight
          property_type:
            primitive_type: DT_DOUBLE

Import Graph Data

To import your data, you need to first bind the data source and then submit a bulk loading job.

Note

@ means the file is a local file and need to be uploaded.

gsctl create datasource -f ./import.yaml -g <graph_id>

# cat import.yaml
vertex_mappings:
  - type_name: person
    inputs:
      - "@/path/to/person.csv"
    column_mappings:
      - column:
          index: 0
          name: id
        property: id
      - column:
          index: 1
          name: name
        property: name
      - column:
          index: 2
          name: age
        property: age
edge_mappings:
  - type_triplet:
      edge: knows
      source_vertex: person
      destination_vertex: person
    inputs:
      - "@/path/to/person_knows_person.csv"
    source_vertex_mappings:
      - column:
          index: 0
          name: person.id
        property: id
    destination_vertex_mappings:
      - column:
          index: 1
          name: person.id
        property: id
    column_mappings:
      - column:
          index: 2
          name: weight
        property: weight
        
# create data loading job
gsctl create loaderjob -f ./job_config.yaml -g <graph_id>

# cat job_config.yaml
loading_config:
  import_option: overwrite
  format:
    type: csv
    metadata:
      delimiter: "|"
      header_row: "true"

vertices:
  - type_name: person

edges:
  - type_name: knows
    source_vertex: person
    destination_vertex: person