Create AI-task

Creating an AI task.

This method allows you to create an AI task for VOD video processing:

• ASR: Transcribe video
• ASR: Translate subtitles
• CM: Sports detection
• CM: Not Safe For Work (NSFW) content detection
• CM: Soft nudity detection
• CM: Hard nudity detection
• CM: Objects recognition (soon)

How to use:

• Create an AI task, specify algorithm to use
• Get task_id
• Check a result using .../ai/tasks/{task_id} method

For more detailed information, see the algorithm-specific sections below.

AI Automatic Speech Recognition (ASR)

AI is instrumental in automatic video processing for subtitles creation by using Automatic Speech Recognition (ASR) technology to transcribe spoken words into text, which can then be translated into multiple languages for broader accessibility.

Categories:

• transcription – to create subtitles/captions from audio in the original language.
• translation – to translate subtitles/captions from the original language to 99+ other languages.

AI subtitle transcription and translation tools are highly efficient, processing large volumes of audio-visual content quickly and providing accurate transcriptions and translations with minimal human intervention. Additionally, AI-driven solutions can significantly reduce costs and turnaround times compared to traditional methods, making them an invaluable resource for content creators and broadcasters aiming to reach global audiences.

Example response with positive result:

{
  "status": "SUCCESS",
  "result": {
    "subtitles": [
      {
          "start_time": "00:00:00.031",
          "end_time": "00:00:03.831",
          "text": "Come on team, ..."
      }, ...
    ]
    "vttContent": "WEBVTT\n\n1\n00:00:00.031 --> 00:00:03.831\nCome on team, ...",
    "concatenated_text": "Come on team, ...",
    "languages": [ "eng" ],
    "speech_detected": true
    }
  }, ...
}

AI Content Moderation (CM)

The AI Content Moderation API offers a powerful solution for analyzing video content to detect various categories of inappropriate material. Leveraging state-of-the-art AI models, this API ensures real-time analysis and flagging of sensitive or restricted content types, making it an essential tool for platforms requiring stringent content moderation.

Categories:

• nsfw: Quick algorithm to detect pornographic material, ensuring content is “not-safe-for-work” or normal.
• hard_nudity: Detailed analysis of video which detects explicit nudity involving genitalia.
• soft_nudity: Detailed video analysis that reveals both explicit and partial nudity, including the presence of male and female faces and other uncovered body parts.
• sport: Recognizes various sporting activities.

The AI Content Moderation API is an invaluable tool for managing and controlling the type of content being shared or streamed on your platform. By implementing this API, you can ensure compliance with community guidelines and legal requirements, as well as provide a safer environment for your users.

Important notes:

• It’s allowed to analyze still images too (where applicable). Format of image: JPEG, PNG. In that case one image is the same as video of 1 second duration.
• Not all frames in the video are used for analysis, but only key frames (Iframe). For example, if a key frame in a video is set every ±2 seconds, then detection will only occur at these timestamps. If an object appears and disappears between these time stamps, it will not be detected. We are working on a version to analyze more frames, please contact your manager or our support team to enable this method.

Example response with positive result:

{
  "status": "SUCCESS",
  "result": {
      "nsfw_detected": true,
      "detection_results": [ "nsfw" ],
      "frames": [
          {
              "label": "nsfw",
              "confidence": 1.0,
              "frame_number": 24
          },...
      ]
  }
}

Additional information

Billing takes into account the duration of the analyzed video. Or the duration until the stop tag(where applicable), if the condition was triggered during the analysis.

The heart of content moderation is AI, with additional services. They run on our own infrastructure, so the files/data are not transferred anywhere to external services. After processing, original files are also deleted from local storage of AI.

Read more detailed information about our solution, and architecture, and benefits in the knowledge base and blog.

Algorithm-specific details

Create AI ASR task

Transcribing is the process of writing down the words you hear in an audio. Our solution allows you to transcribe audio from your video and get subtitles automatically. To do this, we use modern AI models.

The result:

• Transcription – subtitles in the original language. I.e. audio is in English – subtitles are in English too, audio is in German – subtitles are in German too.
• Translation – subtitles is translated from the original language to any other language.

How to use?

• Explicit call to this AI method. Applicable for any file stored with us or located on the Internet.
• Standard video upload but with automatic subtitle generation. Look at “VOD uploading”.

What language will the subtitles be in?

You can specify the language explicitly, then it will be used to create subtitles: the source language in the audio, the resulting subtitle language. If this is not set, the system will run auto language identification and the subtitles will be in the detected language. The method also works based on AI analysis.

Additionally, when this is not set, we also support recognition of alternate languages in the video (code-switching). For example, when in a video different speakers speak several languages, or when they switch from their native language to English and back. Thus when you have multiple languages in the video it is better to not specify an “audio_language” otherwise AI may force the system to recognize gibberish.

What can be transcribed?

Service uses additional methods to detect presence of speech in audio track, thus improving the detection of any human conversations:

• Speech of one speaker,
• Speech of several speakers,
• Speech in different languages,
• etc

Restriction on music, lyrics most likely will not be created.

What about translation?

It is also possible to automatically translate from the original language to another you need.

To create a translation, specify the desired language explicitly in “subtitles_language” parameter. Otherwise, the subtitles will be in the original language. Translation into different languages should be done by creating separate tasks.

Use MP4 videos to process. This method is not tied to videos that are stored only in our video hosting (look at how get a link to MP4 rendition), so you can use links to any other external file with HTTP/HTTPS access.

For now, only the first audio track can be processed; later this functionality will be improved to allow to use any. Also, not all language pairs are currently supported. If a language pair is not supported for automatic translation, the task status will be FAILURE with description of the reason. Example: eng => uzb. You can request to add the language pair you need for automatic translation. Contact our support.

Example of modes to transcribe and/or translate:

• Auto language detection: { "url":"..." }
• From German language explicitly : { "url":"...", "audio_language":"ger" }
• From any auto-detected to English language explicitly: { "url":"...", "subtitles_language":"eng" }
• From German language to English language explicitly: { "url":"...", "audio_language":"ger", "subtitles_language":"eng" }

Example of setting a task to process MP4 file (animated gif from above):

curl -L 'https://api.gcore.com/streaming/ai/tasks' \
-H 'Content-Type: application/json' \
-H 'Authorization: APIKey 1234$abcd...' \
-d '{
    "url": "https://demo-files.gvideo.io/apidocs/spritefright-blender-cut30sec.mp4" 
}' 

As described above, transcription is done automatically using AI. Therefore, the quality may differ from a manual transcription by a professional person. If this happens to you, then you can download subtitles and change them in an external editor.

Transcription and translation are 2 different AI tasks:

• Transcription is set only for transcription.
• Translation, if non-original languages are set for translation.

Billing takes into account the duration of the analyzed original video.

The heart for transcribing is the AI model Whisper from OpenAI, with additional optimizations and services. The AI models run on our own infrastructure, so the files/data are not transferred anywhere to external services. After processing, original files are also deleted from local storage of AI.

Read more detailed information about our solution, and architecture, and benefits in the knowledge base and blog.

Create AI CM:nsfw task

This algorithm allows to quickly detect inappropriate content, determining that the content is NSFW (“Not Safe For Work”) or normal. Generic info about all capabilities and limits see in the generic “Content Moderation” method.

What is “Not Safe For Work”?

The algorithm has recognized inappropriate content in a video and it might not be suitable to view in public places. The solution provides its confidence level (in percentage) of how sure it is that the content is NSFW, or it most likely does not contain any sexual or similar content.

Different to soft-nudity-detection and hard-nudity-detection, this model will only check for sensitive material that can be considered not-safe-for-work.

How to use?

Frames within the specified video are analyzed.

Response will contain only frames for which the class nsfw is detected with a confidence of more than 50%.

Example of detected NSFW:

{
  "nsfw_detected": true,
  "detection_results": [ "nsfw" ],
  "frames": [
      {
          "label": "nsfw",
          "confidence": 0.93,
          "frame_number": 1
      },..
  ]
}

Example of a response without detecting inappropriate content:

{
  "nsfw_detected": false,
  "detection_results": [],
  "frames": []
}

Please note that the API only provides a set of data (json) about the objects found, so no video is generated. The demo video video (above ^) was specially created based on json from the API for visual demonstration and better perception of the possibilities.

Create AI CM:hard_nudity task

This algorithm allows to detect explicit nudity of the human body (involving genitals) in a video. Generic info about all capabilities and limits see in the generic “Content Moderation” method.

What is Hard nudity detection?

This method is often used to analyze UGC to determine whether videos can be published to all users, or to prohibit publication due to offensive and inappropriate content.

Objects that can be detected:

• ANUS_EXPOSED
• BUTTOCKS_EXPOSED
• FEMALE_BREAST_EXPOSED
• FEMALE_GENITALIA_EXPOSED
• MALE_BREAST_EXPOSED
• MALE_GENITALIA_EXPOSED

Please note that the number of objects is less than in the soft-nudity-detection. This method works faster and better if only exposed body parts detection is required.

How to use?

The information is returned with the video frame number where it was found and probability of the detected object. Nudity detection is done using AI, so for each object a probability percentage is applied; objects with a probability of at least 30% are included in the response.

Video processing speed is approximately 1:5.

Example of detected nudity or body parts:

{
  "nudity_detected": true,
  "detection_results": [ "MALE_GENITALIA_EXPOSED" ]
  "frames": [
      {
          "confidence": 0.75,
          "frame_number": 35,
          "label": "MALE_GENITALIA_EXPOSED"
      },...
  ]
}

Example response when nudity or body parts were not found:

{
  "nudity_detected": false,
  "detection_results": []
  "frames": []
}

There is no universal recipe under which a video can be considered unacceptable, since different services host different types of videos for different audiences: adult content, children’s content, educational content, etc. You can determine the probability threshold at which you consider a video inappropriate. The easiest option is to run several of your videos and analyze the resulting probability coefficient.

Sometimes a detected object at the beginning of the video immediately makes it clear that there is no need to further analyze the video. For such cases, you can use stop tags. Use parameter “stop_objects” to specify comma separated stop tags. It is also possible to specify % probability threshold value, above which the stop tag will be triggered.

  {
      "url": "...",
      "stop_objects": "MALE_GENITALIA_EXPOSED:0.8,FEMALE_GENITALIA_EXPOSED"
  }

Please note that the API only provides a set of data (json) about the objects found, so no video is generated. The demo video video (above ^) was specially created based on json from the API for visual demonstration and better perception of the possibilities.

Create AI CM:soft_nudity task

This algorithm allows to identify explicit nudity and partial nudity too (including the presence of male and female faces and other uncovered body parts) in a video. Generic info about all capabilities and limits see in the generic “Content Moderation” method.

What is Soft nudity detection?

This method is often used to analyze UGC to determine whether videos can be published to all users, or to prohibit publication due to offensive and inappropriate content.

Objects that can be detected:

• ANUS_COVERED
• ANUS_EXPOSED
• ARMPITS_COVERED
• ARMPITS_EXPOSED
• BELLY_COVERED
• BELLY_EXPOSED
• BUTTOCKS_COVERED
• BUTTOCKS_EXPOSED
• FACE_FEMALE
• FACE_MALE
• FEET_COVERED
• FEET_EXPOSED
• FEMALE_BREAST_COVERED
• FEMALE_BREAST_EXPOSED
• FEMALE_GENITALIA_COVERED
• FEMALE_GENITALIA_EXPOSED
• MALE_BREAST_EXPOSED
• MALE_GENITALIA_EXPOSED

This method allows you to identify faces and other body parts. Used to find complex combinations of what is happening in a video. Please note that the number of objects is more than in the hard-nudity-detection. The method is slower.

How to use?

The information is returned with the video frame number where it was found and probability of the detected object. Nudity detection is done using AI, so for each object a probability percentage is applied; objects with a probability of at least 30% are included in the response.

Video processing speed is approximately 1:5.

Example of detected nudity or body parts:

{
  "nudity_detected": true,
  "detection_results": [ "FACE_FEMALE", "BELLY_COVERED" ]
  "frames": [
      {
          "confidence": 0.82,
          "frame_number": 1,
          "label": "BELLY_COVERED"
      },...
  ]
}

Example response when nudity or body parts were not found:

{
  "nudity_detected": false,
  "detection_results": []
  "frames": []
}

There is no universal recipe under which a video can be considered unacceptable, since different services host different types of videos for different audiences: adult content, children’s content, educational content, etc. You can determine the probability threshold at which you consider a video inappropriate. The easiest option is to run several of your videos and analyze the resulting probability coefficient.

Sometimes a detected object at the beginning of the video immediately makes it clear that there is no need to further analyze the video. For such cases, you can use stop tags. Use parameter “stop_objects” to specify comma separated stop tags. It is also possible to specify % probability threshold value, above which the stop tag will be triggered.

  {
      "url": "...",
      "stop_objects": "BELLY_COVERED:0.9,FEMALE_GENITALIA_COVERED"
  }

Please note that the API only provides a set of data (json) about the objects found, so no video is generated. The demo video video (above ^) was specially created based on json from the API for visual demonstration and better perception of the possibilities.

Create AI CM:sport task

This algorithm allows to identify various sporting activities in a video. Generic info about all capabilities and limits see in the generic “Content Moderation” method.

What is Sports activity detection?

Sports activity detection by AI involves using machine learning and computer vision technologies to automatically identify, analyze, and interpret various activities within sports and generic videos. This can include detecting specific types, actions, events, and moments.

This model operates on a video sequence (and not on images as most of the used computer vision models). Make sure your video is at least 10-15 seconds long.

Sports activities can be detected:

• archery
• arm wrestling
• playing badminton
• playing baseball
• basketball dunk
• bowling
• boxing punch
• boxing speed bag
• catching or throwing baseball
• catching or throwing softball
• cricket
• curling
• disc golfing
• dodgeball
• fencing
• football
• golf chipping
• golf driving
• golf putting
• hitting baseball
• hockey stop
• ice skating
• javelin throw
• juggling soccer ball
• kayaking
• kicking field goal
• kicking soccer ball
• playing cricket
• playing field hockey
• playing ice hockey
• playing kickball
• playing lacrosse
• playing ping pong
• playing polo
• playing squash or racquetball
• playing tennis
• playing volleyball
• pole vault
• riding a bike
• riding or walking with horse
• roller skating
• rowing
• sailing
• shooting goal (soccer)
• skateboarding
• skiing

Use cases:

• Sports leagues and content creators can use AI to monitor UGC for unauthorized publications of their content. This can include detecting specific sporting events or activities that are part of copyrighted content.
• Sports fans often miss live games and rely on highlight reels. AI can automatically detect key moments like goals, touchdowns, or game-winning shots in uploaded UGC videos and compile them into personalized highlight reels.

How to use?

The information is returned with the video frame number where it was found and probability of the detected activity. Identification is done using AI, so for each activity a probability percentage is applied; activities with a probability of at least 30% are included in the response.

Video processing speed is approximately 1:5.

Example of detected sports activity:

{
  "sport_detected": true,
  "detection_results": [ "shooting goal (soccer)" ],
  "frames": [
      {
          "label": "shooting goal (soccer)",
          "frame_number": 98,
          "confidence": 0.99
      },...
  ]
}

Example response when sports activities were not found:

{
  "sport_detected": false,
  "detection_results": []
  "frames": []
}

Please note that the API only provides a set of data (json) about the objects found, so no video is generated. The demo video video (above ^) was specially created based on json from the API for visual demonstration and better perception of the possibilities.