Result in detail

Data for the whole video sorted by player

Team configurations during the video

All rallies detected in the video

Highlights -- most interesing rallies -- found in the video

Heatmap of ball bounces on the court

Array of floor bounces containing position, time, player_id

XY-positions of the ball on each 2D frame throughout the video

XY-court positions of the players throughout the video

Confidence values for the different components of the analysis

Array of thumbnails for each player

Metadata about the analysis run.

Warm-up periods detected in the video.

Internal debug data. Present in the output but not intended for general use.

Tracking id of the player

Array of swings performed by player, see description of Swings object below.

Swing type distribution for the given player. Keys are swing type values as described in the swing_type field.

{
  "fh_overhead": 0.152,
  "fh": 0.204,
  "1h_bh": 0.151,
  "2h_bh": 0.098,
  "other": 0.395
}

Total distance covered throughout the video

Unit: m

The fastest sprint throughout the video

Unit: km/h

Timestamp for fastest sprint event

Unit: s

Player heat map throughout the video.

Assigned score based on the players number of swings, fastest sprint and total covered distance.

Start of swing

Units:

• timestamp: s
• frame_nr: index

{
  "timestamp": 13.73,
  "frame_nr": 412
}

End of swing

Units:

• timestamp: s
• frame_nr: index

{
  "timestamp": 14.73,
  "frame_nr": 442
}

Player tracking id

Unit: scalar

"True" if this is a valid swing, as concluded by analysis of player movements and ball trajectory.

Swings that are not valid will be removed from the delivery in the future. For now they are also returned to help evaluate the quality of the analysis.

"True" if the swing is a serve.

The first swing of a rally almost always has serve=true unless we suspect that the first swing of a rally was not detected.

The type of swing performed, as classified by the machine learning model based on the player's movements.

Example: "fh"

"True" if the swing a volley i.e. there was no bounce before the ball was hit.

Attribute "volley" or "not volley" is not included in the swing type. For example, a forehand ("fh") swing can be a volley or not, depending on the ball trajectory before the swing. Therefore, the swing type (e.g. forehand: "fh") is decided by the machine learning model based on player's movements. The volley attribute is added later based on the ball trajectory and other factors such as distance between the player and the net (a ball bounce at the floor is not always detected correctly, that's why other factors are also included).

"True" if the swing belongs to a rally.

Two timestamps indicating the start and end of the rally the swing belongs to. If is_in_rally is false, this field is irrelevant.

Unit: s
Example:

[
  1.36699,
  5.96699
]

Time of the ball-racket contact

Units:

• timestamp: s
• frame_nr: index

{
  "timestamp": 14.23,
  "frame_nr": 427
}

Machine learning confidence value of swing_type. Values are between 0 and 1, higher values indicate higher confidence that the swing is of the given type.

Unit: scalar [0-1]
Example: 0.45

Machine learning confidence that the detected event is an actual swing.

Unit: scalar [0-1]
Example: 0.97

Overall confidence that the swing is valid i.e. represents a successful attempt to hit the ball. It is calculated by combining machine learning confidences and consistency of the swing time and place with the ball data. Values are between 0 and 1, higher values indicate higher confidence that the swing is valid.

Unit: scalar [0-1]
Example: 0.65

Confidence that the swing is a volley, a float value between 0 and 1. It is an addition to the "volley" attribute, to allow for a more fine-grained analysis. Volley is a boolean attribute that is set to true if the confidence_volley is higher than 0.5.

Confidence_volley is computed based on the floor bounce data, distance between the player and the net, swing type, time after the previous swing, orientation of the player body, serve attribute of the swings.

Unit: scalar [0-1]
Example: 0.79

Position of player on court (X,Y) when the ball was hit, in meters. Defined as the middle point between the player's right foot and left foot.

Unit: m

Distance on image between the player's bounding box and ball in impact frame. The distance is normalized to the average size of the player's bounding box. Distances larger than one usually indicate that it is not a valid swing since the ball was too far away from the player.

Unit: scalar

Estimated ball velocity just after the ball was hit

Unit: km/h

Not in use yet. Location where ball hit the court or where intercepting player returned ball from

Unit: m

Not in use yet. Impact description

Not in use yet. Tracking ID of intercepting player if applicable

Unit: scalar

Not in use yet. Image coordinates of ball after swing

Unit: pixel

2D Pose data for player during swing

Start of team session

Unit: s

End of team session

Unit: s

Array of player IDs in the front team (close to the camera)

Array of player IDs in the back team (far from the camera)

Array of players (in both teams), see description of Players object above.

Player objects here only contain content from within the team session, e.g. activity score is computed based on player's swings, speed and covered distance during the session. They do not include player location heatmap, swing list and swing type distribution.

Units: s
Example:

[
  [1.36699, 5.96699],
  [19.533, 25.9],
  [32.8, 36.533]
]

Start of highlight

{
  "timestamp": 13.73,
  "frame_nr": 412
}

End of highlight

{
  "timestamp": 24.73,
  "frame_nr": 742
}

Type of highlight. So far two types are available: "longest_rally" and "fastest_rally". Longest rally is determined purely by its duration. Fastest rally is determined by the average speed of the ball and also average speed of the players.

Example: "longest_rally"

Array of players that were active during the highlight, see description of Players object above.

Players object here only contain content from within the highlight duration, e.g. only those swings that have been performed during the highlight. It does not contain player location heatmap (because it occupies a lot of space, and probably has little interest for the short duration of the highlight, while the heatmap for the whole video is already present in the player object).

Duration of highlight (end - start)

Unit: s

Total number of swings during the highlight

Unit: scalar

A relative measure of how fast the ball moves across the video frames during the highlight. It is computed from ball apparent positions on the image, rather than it's positions in the 3D world. It is meant to give a rough idea of which rallies involved faster ball movement. Ball speed here shouldn't be confused with the ball speed in the swing object, which reflects the actual speed of the ball after the hit in 3D world, in km/h.

Unit: scalar

Similar to the ball speed defined above. It is a relative measure of the distance travelled by the ball across the video frames during the highlight. It is computed from ball apparent positions on the image, rather than it's positions in the 3D world. It is meant to give a rough idea of which rallies involved more ball movement.

Unit: scalar

Total distance travelled by all the players during the highlight. Computations are based on the player positions in the court coordinates, after proper smoothing.

Unit: m

The player distance defined above divided by highlight duration in seconds. Essentially, the sum of average player speeds during the highlight.

Unit: m/s

A metric used to sort rallies by their intensity and select the "fastest rally" highlights. It is a product of 4 values: ball' and players' speed and distances, where the speeds are taken to the cubic power. In other words, it is higher for rallies with fastest ball and player motion, but slightly lower for very short rallies.

Unit: scalar

2D matrix of integer numbers delivered as an array of arrays. Each numbers indicates the total number of ball bounces in the corresponding cell of the court.

The cell size is 1x1 meter, and the bounce matrix dimensions are 23.77x8.23, which reflect the court dimensions of 23.77x8.23 meters.

Units

• scalar, integer non-negative

Example:

[
  [0, 0, 0, 2, 2, 2, 0, 0, 0, 0],
  [2, 1, 4, 1, 4, 0, 1, 1, 0, 0],
  [2, 1, 1, 4, 1, 2, 2, 0, 0, 0]
]

Units: seconds

Units: meters [0:8.23, 0:23.77]

Units: scalar

Values: "floor" or "swing"

Timestamps along with coordinates (X,Y) of the ball for every frame of the video where a ball was detected. The coordinates are float numbers in the range [0,1] where (0,0) is the top left corner of the frame and (1,1) is the bottom right corner.

Results for one frame are given as an object with the following properties: timestamp, X, Y. These objects for different frames are combined in an array.

For some frames there is no ball detected, then the difference between timestamps in subsequent array elements is large.

Units

• timestamp: seconds
• X, Y: scalar [0-1]

Example:

[
  {
    "timestamp": 0.03301,
    "X": 0.70365,
    "Y": 0.66944
  },
  {
    "timestamp": 0.5,
    "X": 0.69792,
    "Y": 0.66111
  }
]

Timestamps and image coordinates (X,Y) and court positions (X,Y) for every player throughout the video. Data are given at approximately 6 frames per second (data are sampled sparser than typical 30 fps for video recording to save on output size).

The image coordinates are float numbers in the range [0,1] where (0,0) is the top left corner of the frame and (1,1) is the bottom right corner. The image coordinates correspond to the center of the player bounding box.

The court positions are distance, in meters, from top left corner of the court (as seen from camera).

Results for one frame are given as an object with the following properties: timestamp, X, Y. 'court_X', 'court_Y'. These objects for different frames are combined in a list for each player. These lists are delivered as a dictionary, with keys being player IDs.

When the player is outside of camera view, or outside of the court, there is no data delivered.

Units

• timestamp: seconds
• X, Y: scalar [0-1]
• court_X, court_Y: scalar [0:8.23, 0:23.77]

Example:

{
  "161": [
    {
      "timestamp": 0.33,
      "X": 0.30365,
      "Y": 0.26944,
      "court_X": 1.41,
      "court_Y": 2.55
    },
    {
      "timestamp": 0.5,
      "X": 0.39792,
      "Y": 0.29111,
      "court_X": 1.49,
      "court_Y": 3.1
    }
  ]
}

Confidence level in the poses detected for each player. Object properties are player IDs, the values are objects with two properties ‘mean’ (mean confidence) and ‘count’ (over how many poses this mean value was computed).

Units

• mean: scalar [0-1]
• count: integer

Example:

{
  "1995": {
    "mean": 0.66623,
    "count": 75873
  },
  "2631": {
    "mean": 0.64581,
    "count": 81957
  },
  "2854": {
    "mean": 0.63266,
    "count": 84468
  },
  "3031": {
    "mean": 0.67041,
    "count": 79622
  }
}

Ball detection frequency is a relative fraction of video frames where a ball was detected. Ball detection confidence is a measure of confidence that the determined ball location is correct

Units

• ball_detection_frequency: scalar [0-1]
• ball_detection_confidence: scalar [0-1]

Example:

{
  "ball_detection_frequency": 0,
  "ball_detection_confidence": 0.30984
}

Object properties are player IDs, the values are objects with properties mean (mean confidence), count (over how many occurrences this mean value was computed), and ball_nearby (how often the detected swinging movements matches a ball detection nearby)

Units

• mean: scalar [0-1]
• ball_nearby: scalar [0-1]
• count: integer

Example:

{
  "1995": {
    "mean": 0.66621,
    "ball_nearby": 0.71107,
    "count": 488
  },
  "2631": {
    "mean": 0.67024,
    "ball_nearby": 0.62647,
    "count": 597
  },
  "2854": {
    "mean": 0.65375,
    "ball_nearby": 0.6691,
    "count": 547
  },
  "3031": {
    "mean": 0.52166,
    "ball_nearby": 0.51696,
    "count": 619
  }
}

Confidences averaged over all players (proportionally to the corresponding counts): object with properties pose, ball, swing (computed from mean in swing confidences), swing_ball (computed from ball_nearby in swing confidences). The final confidence is average of all others.

Units

• pose: scalar [0-1]
• swing: scalar [0-1]
• swing_ball: scalar [0-1]
• ball: scalar [0-1]
• final: scalar [0-1

Example:

{
  "pose": 0.65326,
  "swing": 0.6245,
  "swing_ball": 0.62506,
  "ball": 0.30984,
  "final": 0.55316
}

Each player ID is mapped to an array of the top 5 thumbnail frames for that player. Each thumbnail comes in 4 variants which differs in how close the crop is to the players face. The smallest frames the head, the medium sized frames the head and some of the chest, the large box frames the head and torso and the largest frames the full person.

Boundingboxes follow the (xmin, ymin, xmax, ymax) format with coordinates normalized to image dimensions.

Example:

{
"4": [
    {
        "bbox": [
            [
                0.6133851574652889,
                0.11593489143187785,
                0.6234476996666691,
                0.13102870473394823
            ],
            [
                0.6083538863645988,
                0.11090362033118772,
                0.6284789707673593,
                0.14109124693532848
            ],
            [
                0.6083538863645988,
                0.10889111189091168,
                0.6284789707673593,
                0.16926636509919316
            ],
            [
                0.6076457500457764,
                0.11096051335334778,
                0.624550998210907,
                0.21223686635494232
            ]
        ],
        "frame_nr": 411,
        "timestamp": 13.699999809265137,
        "score": 0.3808106226830157
    },
    {
        "bbox": [
            [
                0.6137423031787128,
                0.11565495662491035,
                0.6239184624691754,
                0.1309191955606041
            ],
            [
                0.6086542235334815,
                0.11056687697967908,
                0.6290065421144067,
                0.14109535485106664
            ],
            [
                0.6086542235334815,
                0.10853164512158658,
                0.6290065421144067,
                0.16958860086436167
            ],
            [
                0.6079655289649963,
                0.1108631119132042,
                0.6248025298118591,
                0.21169747412204742
            ]
        ],
        "frame_nr": 412,
        "timestamp": 13.73330020904541,
        "score": 0.3777012344630819
    },
    {
        "bbox": [
            [
                0.6054023159644926,
                0.1216291664145315,
                0.6161930309631503,
                0.13781523891251796
            ],
            [
                0.6000069584651637,
                0.11623380891520267,
                0.6215883884624791,
                0.14860595391117562
            ],
            [
                0.6000069584651637,
                0.11407566591547114,
                0.6215883884624791,
                0.17881995590741703
            ],
            [
                0.6001584529876709,
                0.11747521162033081,
                0.6205766797065735,
                0.2218719720840454
            ]
        ],
        "frame_nr": 82,
        "timestamp": 2.733330011367798,
        "score": 0.3621332238345576
    },
    {
        "bbox": [
            [
                0.6143657415112168,
                0.11661404361330573,
                0.6246624738971084,
                0.13205914219214313
            ],
            [
                0.609217375318271,
                0.11146567742035994,
                0.6298108400900542,
                0.1423558745780347
            ],
            [
                0.609217375318271,
                0.10940633094318161,
                0.6298108400900542,
                0.17118672525853115
            ],
            [
                0.6088064908981323,
                0.11195679754018784,
                0.625842809677124,
                0.21372298896312714
            ]
        ],
        "frame_nr": 413,
        "timestamp": 13.76669979095459,
        "score": 0.35957940319590875
    },
    {
        "bbox": [
            [
                0.6165482997894287,
                0.1194416880607605,
                0.6274006366729736,
                0.13572019338607788
            ],
            [
                0.6111221313476562,
                0.11401551961898804,
                0.6328268051147461,
                0.1465725302696228
            ],
            [
                0.6111221313476562,
                0.11184505224227906,
                0.6328268051147461,
                0.1769590735435486
            ],
            [
                0.6106997132301331,
                0.11665115505456924,
                0.6285014748573303,
                0.2224864512681961
            ]
        ],
        "frame_nr": 419,
        "timestamp": 13.966699600219727,
        "score": 0.35655416500383064
    }
]
}

Detected sport type used for the analysis.

Example: "tennis_doubles"

Sport type as originally provided before any internal reclassification.

Example: "tennis_unknown"

If true, the output contains only swings that belong to a rally.

Technical information about the input video.

Number of unique players tracked.

Number of rallies detected.

Number of team sessions detected.

Number of floor ball bounces detected.

Start of the warm-up period.

Unit: seconds

End of the warm-up period.

Unit: seconds

Confidence that this period is a warm-up.

Unit: scalar [0-1]

List of rally time intervals detected within the warm-up period, each as [start_time, end_time].

Unit: seconds

Method used to detect the warm-up.

Example: "multiballs"

Reason the period was classified as a warm-up.

Example: "multi_ball_detection"